We are still working on this KnowledgeBase!
Cool Data Mover Manual
Part 0 – Cool Data Mover Concepts
Based on extensive knowledge of system development with the OutSystems Platform and database management systems CoolProfs has developed a unique solution to stage application data.
The Cool Data Mover allows you to easily automate the transfer of data between environments on any platform installation (PaaS, On Premise or Hybrid).
The advantages are:
- Direct connection to the OutSystems meta model
- Easily define application datasets to stage
- Single click export, delete and import of data. Fully GDPR compliant
Reduce costs
- Define a data package once, stage it many times
- Single click export, delete and import data
- Less database storage needed in non-production environments by using filters
Improve quality
- Direct connection to OutSystems meta model
- Automatic validation and data consistency check
- Filters to export subsets of data
Dictionary
| Term | Explanation |
|---|---|
| Anonymize values | Special anonymize functions are available for data operation Export. Per attribute of a selected entity in a data package, it is possible to switch-on or switch-off data anonymization for that attribute. When switched-on, the type of anonymization must be selected. The data type of an attribute determines which types of anonymization are possible. For example, datatype ‘string’ offers a number of text anonymizations such as Random Phone, Random text, Random URL, etc. Datatype ‘date’ offers only one date anonymization where the min and max date can be defined. Anonymize values exist on two levels: • Generic level – anonymize values are set on the highest level. This level is used to default the anonymize values on data package level. • Data package level – anonymize values are set on data package level (lowest level). This level supersedes the values that are set on generic level. By default the anonymize values of generic level are inherited but it is possible to overwrite these values. |
| Custom Anonymize | On special occasions where you want to use a tailormade anonymization function, it is possible to turn-on the Custom Anonymize switch. If turned-on, a custom anonymization webservice will be called upon export. This enables you to implement your own anonymization logic in the CDM_CustomAnonymization module. After the custom anonymization, the attribute specific anonymization option will be applied as well. |
| Data operation | A data operation always takes place on the database of the ‘local’ environment (i.e. the environment where the data operation executes). |
| DELETE | Deletion of records of entities that are defined in the data package. A delete operation takes place on the ‘local’ environment (i.e. the environment where the delete operation is executed). Notice that data is never deleted from Lookup and Static entities and also data is never deleted from the OutSystems system entities in the ServiceCenter module. |
| EXPORT | Export of records of entities that are defined in the data package. The results of the export operation are stored on selected storage location. |
| IMPORT | Import of records of entities that are defined in the data package. The import files are taken from the selected storage location. |
| Data package | A data package contains the definitions/settings of which data operations are possible on what data (one or more entities). There are three types of data operations: Export, Import and Delete. Depending on which data operations are applicable for the package, specific settings related to the type of operation can be defined. Two filter options are available for data operation Export. Custom filters can be defined within the data package if data operation Export is selected and if the custom filter option is switched on. The tenant filter option is available within Export or Delete packages and allows you to filter the data based on a selected tenant. When exporting data, it is possible to anonymize the data while exporting it. Within a data package you can configure your desired anonymizations which affects your export data elements. |
| Import behavior of Move entities | This data package IMPORT setting can be used to define the behavior of move entities during the import process. There are two possible situations when importing data from the source to the target environment: 1. A record of the entity in the Source environment does not exist in the Target environment. 2. A record of the entity in the Source environment already exists in the Target environment. Situation 1 In this case, the CDM inserts a new record in the Target environment. There are no alternatives to process this situation. Situation 2 In this case, you can influence the behavior of the import process in two ways: a. Perform a lookup in the target environment and use the existing record when a match is found (i.e. do NOT insert a new record). b. Do not perform a lookup and ALWAYS insert a new record. This causes duplicates of records. |
| Move entity | An entity that is marked in the data package to be moved from a source environment to a target environment. Rules: 1. Entity must exist in target environment. If the entity does not exist in the target environment, then a warning will be displayed to the user indicating the data models in the source and target environment are incompatible. 2. Records that do not exist in the target environment will be copied to the entity in the target environment. 3. Records that already exist in the target environment will be copied to or reused in the target environment, depending on the IMPORT setting “Import behavior of Move entities”. |
| Static entity | A Static entity is a special kind of entity. Like lookup entities the data is referenced rather than moved. This implies that the content of these entities must be the same (same records in Source environment and Target environment). A static entity will be recognized automatically by the Datamover as it is defined as a static entity in OutSystems. Therefore it automatically knows how to match records between source and target environment and therefore it is not possible to define match attributes for static entities. Rules: 1. Same rules apply for Static entities as for Lookup entities. |
| Match attribute | To identify if records already exist in the target environment, match attributes are used to match records of the entity in the source environment to those of the entity in the target environment. A record is uniquely matched if and only if all match attributes in the source environment have got the same value in the destination environment. |
| Soft key attribute | When a text or (long)integer attribute refers to (the key of) an entity, but is not defined as a foreign key in Outsystems. This means that no constraints will exist in the database to enforce referential integrity. In the CDM the possibility exists to mark an attribute as a soft key. This means the CDM will treat that attribute as if it was an optional, ignorable foreign key. |
| Source environment | Environment where data is sourced from when moving data from one environment to another. |
| Target environment | Environment where data is moved to when moving data from one environment to another. |
Part 1 – DATA PACKAGES
A data package is the definition of WHAT data is affected by one of the data operations of the CDM. Possible operations on a data package are listed in the table below.
| Data Operations *) | Description | Remarks |
|---|---|---|
| Export | To export data from entities within the data package, a predefined data package must be selected in the source environment. The data will be exported to the location specified in the package settings. | When exporting data, optional filters can be defined to limit the amount of data exported. GDPR compliancy can be enforced by switching on ‘Apply anonymization during export’. Details need to be refined per entity per attribute. |
| Delete | To empty (clean) data from entities within the data package. Only data within the definition of the data package will be deleted. | No data is deleted from Lookup and Static entities. No data is deleted from the standard OutSystems system entities (entities in the ServiceCenter module). |
| Import | To import data into entities within the data package from files which are created by an export operation. The data will be imported from the import location as specified in the package settings | Before data is inserted, validations are performed to ensure unique matches for Lookup and Static entities are available. |
*) Site properties are available to prevent accidental exports, imports or deletes. Only the system administrator can change these settings in Service Center. See Applications, modules and site properties.
Data Package overview
Data Package details
Package name / description
The name and description of the package. The description will be displayed as tooltip in the list of packages when hoovering over the package icon.
Data operations
Specifies which operations are allowed using this package (might be disabled based on Service Center site properties).
| Allow custom filtering | Enables the possibility to define and use filters in export packages. |
| Allow tenant filtering | Enables the options to use a tenant filter in export- or delete packages. |
| Max processes | Defines the maximum number of light-BPT processes to be started at once. |
| Rows per file | Specifies the number of rows that wil be stored inside one file. |
| Skip binaries over | When a binary exceeds this size the binary will be replaced with a dummy binary. |
| Max MB to process per worker | Specifies the max MB’s of binary data that one worker will process. |
| Max MB to process in parallel | Specifies the max MB’s of binary data that all workers will process at once. |
Data Package details – Import settings
Import from location / directory
The location (as specified in the Storage Locations) and directory from which the data files will be imported.
Rows per process
The amount of rows that will be processed per import process. This number must be smaller than or equal to the amount of rows in a single package file, as defined on the export package.
Import behavior of Move entities
The behavior when importing Move entities.
- Perform a lookup – Try to find an existing record (based on the match attributes) and use that record when one is found. Only when no match is found a new record will be created.
- Do not perform a lookup – No attempt is made to find an existing record and a new record is always created.
Update upon match
When an existing record is found, update the non-match attributes
Suspend BPT processes during import
Suspend all BPT processes that are triggered upon a Create (insert) or Update of a Move entity record. Usually the triggered processes will update the database. Since that data will also be imported there is no need for the processes to run. This also avoids the triggering of possible millions of processes when importing that amount of records.
Convert to local timezone
Convert date time values to current database time zone on import. Example:
| Exported | UTC | 17-6-2020 07:00:00 |
| Import | UTC +3 | 17-6-2020 10:00:00 |
| Import | UTC -8 | 16-6-2020 23:00:00 |
Data Package details – Export settings
Export to location / directory
The location (as specified in the Storage Locations) and directory to which the data files will be written
Export to directory
The location (relative to the ‘Export server’) where the data files will be placed. Use UNC notation or specify a directory on a network-share.
Apply anonymization
Indicates whether or not the export data will be anonymized. Specifying the anonymization per attribute needs to be done separately in order to determine which anonymization algorithm to use.
Data Package details – Actions
Save/Cancel
The changes in the package will be saved or rolled back.
Delete
The package will be deleted; existing monitoring data will be kept.
Selecting Modules
- Selected
This column shows if there are entities selected within this module where we want to perform data operations on. - Module
Click on a module to see the entities of the module and which entities are selected to perform the desired data operations. - Upload definition
(see below) - Download definintion
In most cases, you will first make a data package to export data in the source environment. Then you will make a data package in the target environment to import the data. Because the export and import are referring to the same dataset, you can use the data package definitions of the export for the import. First download the definition in the source environment and thereafter upload definition in the target environment allows you to copy the data package definitions from the source to the target environment. - Filter
Use this filter to quickly locate the module
Selecting Entities
- Show or hide validation messages.
- Revalidate Package. This function can be used when the data package is outdated compared to the meta model of Outsystems.
- Select the entities that need to be moved between environments or need to be deleted from this environment.
- Symbolizes the type of entity (move or lookup). A special type of lookup entities are static entities which can be recognized by the yellow items within the symbol.
- Hover to show details or expand to show attributes of the entity.
- Define lookup or move entity. Lookup entities are assumed to be present in the target environment and need to have match attributes to be able to reference them. Move entities to be moved to the target environment.
- When selected, the custom anonymization webservice will be called upon export. This enables you to implement your own anonymization logic in the CDM_CustomAnonimization module. After the custom anonymization, the attribute specific anonymization option will be applied as well.
- Refresh entity definitions. This function can be used when the data definition(s) are outdated compared to the OutSystems metamodel.
Selecting Entities – Validations
- Export validation (see below)
- Delete validation
Based on the OutSystems metamodel the Data Mover checks/validates the data operations that are defined for this data package. Notice there are no import validations at this stage. Import validations are done when actually performing the import process.
- Blocking error messages
These messages are blocking, i.e. these messages must be resolved. If not, you cannot excecute the related data operation. So for example if you want to export entity Booking, the Data Mover automatically detects a relationship between Booking and Schedule. To resolve this dependency, you can include entity Schedule in your set with entities. Or you can skip the Schedule attribute in entity Booking. Notice that the Schedule attribute must be optional to be able to skip it! - Non-blocking warning messages
These messages are not blocking because there are no conflicts at this moment in the datamodel. Based on the datamodel these conflicts could emerge. In this example, we want to perform a delete operation on entity Booking. Since this entity is related to entity Paxbooking, you should also delete the related paxbookings. Since entity Paxbooking is empty (appearently the foreignkey in entity Paxbooking is optional) there is no physical relationship between Booking and Paxbooking and therfore you can delete bookings without having Paxbookings in your set with entities. Still it is better to include Paxbookings in your set of entities to prevent an errormessage in the future.
Selecting Match attributes
- Match (import data)
When running an import, there are two situations in which you need to match data with the existing data in the target environment:
- When importing lookup entities. In this situation, no data will be inserted, but instead the data will be matched with the existing data in the target environment. The match is performed on all attributes marked as match. If a match is found, that record will be used. If no match is found, the import process stops with an error message.
- When importing move entities with import behavior set to “Perform a lookup”. In this situation, if a match is found, that record will be used. If no match is found, the import process continues and the record will be inserted.
- Indicates match status of entity:
- Missing attributes: no match attribute have been defined.
- Validate match attributes: match attributes have been defined but we need to check whether these uniquely identify records.
- Match attributes uniquely identify entity records: match attributes have been defined and tested correctly.
Important!!
Make sure there is an index containing all match attributes in the target environment. If not, this could have a severe performance impact.
Attributes
- Skip (export data)
Use this option to skip attributes when running an export. This will result in an empty column upon import. This option can only be used with non-mandatory attributes. - Anonymize (export data)
Check this field when you want to anonymize the data of this attribute during export. Primary and foreign keys cannot be anonimized. When checked, you will have to choose which anonimization is applicable (see “Anonimization options”). - Replace (export data)
Use this option to replace the value of the attribute with a specified value when running an export.
Attributes – Soft keys
- Convert attribute to softkey
Click the icon of a text or (long) integer attribute to convert it into a softkey or to remove the softkey settings. The popup will appear. - Is soft key
Indicates whether this attribute is a soft key or not. - Destination entity
The entity to which this softkey refers to.
When a attribute is defined as being a softkey, the CDM treats it exactly the same as if it were defined as a ‘normal’ foreign key in OutSystems, meaning key translation will take place.
Anonimization options
- String
For text based attributes there is a long list of options which can be chosen to anonymize GDPR sensitive data. They can either be totally scrambled of contain certain presets. - Decimal
For attributes containing decimals, like prices. - Date
For attributes containing a date or datetime - Integer
For attributes that contain round numbers.
Based on the selected option presets will be shown to further customize the randomization.
Part 2 – DATA operations
Please contact CoolProfs for the rest of this manual! We’re still working on the KnowledgeBase version here!