Creating Retention Policies

Lavatory policies are implemented as Python plugins. Each policy is a .py file named after an Artifactory repository.

Each plugin represents one repository. The file name should match the repository name, replacing - with _.

For example, the repository yum-local should have a retention policy named yum_local.py

Anatomy of a Policy

Each policy needs to provide one function, purgelist(). This function takes one argument, artifactory, which is an instance of the lavatory.utils.artifactory.Artifactory class. This argument handles all communication with artifactory.

This function needs to return a list of artifacts to delete.

Docstring Description

The docstring following the function definition will be used as the policy description. This gets used in logging, as well as generating a list of all active policies.

Return Value

The return value of the policy should be a list of artifacts to delete. The artifacts are a dictionary that at minimum needs a path and name key. These keys are used by the delete function to remove the artifact.

path: path to artifact in the repository

name: Name of the artifact

Example Minimal Return:

[{ 'path': '222', 'name': 'Application-10.6.0-10.6.0.07-9cd3c33.iso'}]

This will delete artifact <repo_name>/222/Application-10.6.0-10.6.0.07-9cd3c33.iso

Policy Helpers

Below are some helper functions to assist in writing policies. These include easy ways to do time-based retention, count-based retention, or searching with AQL.

Time Based Retention

This policy will purge any artifact in the repository older than 120 days.

def purgelist(artifactory):
    """Policy to purge all artifacts older than 120 days"""
    purgable = artifactory.time_based_retention(keep_days=120)
    return purgable

Artifactory.time_based_retention(keep_days=None, time_field='created', item_type='file', extra_aql=None)

Retains artifacts based on number of days since creation.

extra_aql example: [{“@deployed”: {“$match”: “dev”}}, {“@deployed”: {“$nmatch”: “prod”}}] This would search for artifacts that were created after <keep_days> with property “deployed” equal to dev and not equal to prod.

Parameters:

• keep_days (int) – Number of days to keep an artifact.
• time_field (str) – The field of time to look at (created, modified, stat.downloaded).
• item_type (str) – The item type to search for (file/folder/any).
• extra_aql (list) –

Returns:

List of artifacts matching retention policy

Return type:

list

Count Based Retention

This policy will retain the last 5 artifacts of each project in a repository.

def purgelist(artifactory):
    """Policy to keep just the 5 most recent artifacts."""
    purgable = artifactory.count_based_retention(retention_count=5)
    return purgable

Artifactory.count_based_retention(retention_count=None, project_depth=2, artifact_depth=3, item_type='folder', extra_aql=None)

Return all artifacts except the <count> most recent.

Parameters:

• retention_count (int) – Number of artifacts to keep.
• project_depth (int) – how far down the Artifactory folder hierarchy to look for projects.
• artifact_depth (int) – how far down the Artifactory folder hierarchy to look for specific artifacts.
• item_type (str) – The item type to search for (file/folder/any).
• extra_aql (list) –

Returns:

List of all artifacts to delete.

Return type:

list

AQL Filtering

You can also use AQL to search for artifacts if you need more control than the count-based retention or time-based retention helps.

def purgelist(artifactory):
    """Policy to purge artifacts with deployed property of dev and not prod."""
    aql_terms = [{"@deployed": {"$match": "dev"}}, {"@deployed": {"$nmatch": "prod"}}]
    extra_fields = ['property.*']
    purgable = artifactory.filter(terms=aql_terms, fields=extra_fields, depth=None, item_type="any")
    return purgable

All of the terms in aql_terms will be ANDed together and searched.

The above policy would use the below full AQL to search for artifacts.

items.find({"$and": [{"@deployed": {"$match": "dev"}},
           {"@deployed": {"$nmatch": "prod"}}, {"path": {"$nmatch": "*/repodata"}},
           {"repo": {"$eq": "yum-local"}}, {"type": {"$eq": "any"}}]}).include("stat", "property.*")

Artifactory.filter(terms=None, depth=3, sort=None, offset=0, limit=0, fields=None, item_type='folder')

Get a subset of artifacts from the specified repo. This looks at the project level, but actually need to iterate lower at project level

This method does not use pagination. It assumes that this utility will be called on a repo sufficiently frequently that removing just the default n items will be enough.

Parameters:

• terms (list) – an array of jql snippets that will be ANDed together
• depth (int, optional) – how far down the folder hierarchy to look
• fields (list) – Fields
• sort (dict) – How to sort Artifactory results
• offset (int) – how many items from the beginning of the list should be skipped (optional)
• limit (int) – the maximum number of entries to return (optional)
• item_type (str) – The item type to search for (file/folder/any).

Returns:

List of artifacts returned from query

Return type:

list