Data Purge
Data Purge
Overview
There is a possibility to delete completed run after a period of days. Here are the CS Configuration parameters that configure this job:
Parameter name | Parameter type | Default value | Scope | Description |
|---|---|---|---|---|
| data_purge_keep_days | number >= 0 | 90 | CS | The maximum number of days to keep data. Elasticsearch is configured to keep indexes no more that 125 day. |
| data_purge_purge_interval_days | number >= 1 | 1 | CS | The period in days to perform data purge (i.e. purge data every N days and remove all runs that older data_purge_keep_days). The data_purge_purge_interval_days must be < data_purge_keep_days |
| data_purge_connection_timeout_min | number > 1 | 10 | CS | Connection timeout for HTTP request in minutes. Depends on data purge size. Should be increased if DataPurge logs contains timeout errors. |
The system job runs every data_purge_purge_interval_days at midnight and removes all:
- runs
- logs
- activity logs
- CS storage resources (unused ones)
- deleted users
that created older than data_purge_keep_days ago.
The storage purge removes everything from bucket data and started from /document_set/ path that doesn't belongs to existing document set. So do not put you information here.
Data Purge logs
To audit the data purge job log files, use one of the search filter:
- thread:"DataPurge-1"
- logger:"eu.ibagroup.easyrpa.cs.service.impl.DataPurgeServiceImpl"
Metrics Data Purge
Metrics are stored in InfluxDB, and data is purge via retention policy. By default it is 90 days.
You can change it. For this you need login into InfluxDB administrator console and update retention policy for easyrpa bucket.
Application server post-installation
System | Access Type | Access URL | User name | Password |
|---|---|---|---|---|
| InfluxDB | Public | https://<app_server_host>:8446/ | admin | $INFLUXDB_ADMIN_PASSWORD |
Call Data Purge using CS API
There is a data-purge-controller in Control Server API that allows to run the data-purge by a REST call.
PUT /api/v1/data_purge
The method allows to specify the keepDays parameter and purge domains:
- activity - activity Log records from javers tables older than keepDays
- run - purge AP Runs and mongoDB history older than keepDays
- log - platform logs older than keepDays
- metric - InfluxDB Metrics older than keepDays. Perform direct InfluxDB RERST call that could be failed because of too long response for specified timeout
- object_storage - MongoDB history records using runs older than keepDays, but not runs itself
- storage - storage folders, HTT, model trainings data, data/document_set unused (by exist DocumentSet) folders
- users - purge deleted users (platform uses soft delete, the operation tries to perform the hard one)