Module continuous_delivery_scripts.utils.configuration

Utilities in charge of fetching configuration values for the ci scripts.

Classes

class ConfigurationVariable (*values)

Expand source code

class ConfigurationVariable(enum.Enum):
    """Project's configuration variables."""

    PROJECT_ROOT = 1
    """Relative path to the root of the project from pyproject.toml."""
    PROJECT_CONFIG = 2
    """Relative path to the file comprising project's configuration."""
    NEWS_DIR = 3
    """Relative path to the directory comprising news files."""
    VERSION_FILE_PATH = 4
    """Relative path to the file comprising project's version."""
    CHANGELOG_FILE_PATH = 5
    """Relative path to the changelog file."""
    MODULE_TO_DOCUMENT = 6
    """Name of the module to generate documentation for."""
    DOCUMENTATION_DEFAULT_OUTPUT_PATH = 7
    DOCUMENTATION_PRODUCTION_OUTPUT_PATH = 8
    """Relative path to the folder where documentation will be generated."""
    GIT_TOKEN = 9
    """GIT token allowing committing back to the repository."""
    BETA_BRANCH = 10
    """Name of beta/QA branch."""
    MASTER_BRANCH = 11
    """Name of the main/production branch."""
    RELEASE_BRANCH_PATTERN = 12
    """Pattern of release branches."""
    REMOTE_ALIAS = 13
    """Git's remote alias."""
    LOGGER_FORMAT = 14
    """Tools' logger format."""
    BOT_USERNAME = 15
    """Username of the Git bot account."""
    BOT_EMAIL = 16
    """Email of the Git bot account."""
    ORGANISATION = 17
    """Name of the organisation managing the project."""
    ORGANISATION_EMAIL = 18
    """Email of the organisation managing the project."""
    AWS_BUCKET = 19
    """AWS bucket to use if needed."""
    PROJECT_NAME = 21
    """Name of the project."""
    PROJECT_UUID = 22
    """UUID of the project to use in SPDX reports."""
    PACKAGE_NAME = 23
    """Name of the package."""
    SOURCE_DIR = 24
    """Relative path to the source of the project."""
    IGNORE_REPOSITORY_TEST_UPLOAD = 25
    """States whether to release package to a test registry (e.g. pypi test) test before release."""
    FILE_LICENCE_IDENTIFIER = 26
    """SPDX identifier for the project's licence."""
    COPYRIGHT_START_DATE = 27
    """Project's copyright start year."""
    ACCEPTED_THIRD_PARTY_LICENCES = 28
    """List of accepted 3rd party licences for dependencies."""
    PACKAGES_WITH_CHECKED_LICENCE = 29
    """List of dependencies whose licence was manually checked."""
    PROGRAMMING_LANGUAGE = 30
    """Project's programming language."""
    AUTOGENERATE_NEWS_FILE_ON_DEPENDENCY_UPDATE = 31
    """States whether news file should be generated on dependency updates."""
    DEPENDENCY_UPDATE_BRANCH_PATTERN = 32
    """Pattern of branches comprising dependency updates (e.g. dependabot)."""
    DEPENDENCY_UPDATE_NEWS_MESSAGE = 33
    """Pattern of the news message for dependency update."""
    DEPENDENCY_UPDATE_NEWS_TYPE = 34
    """News file type for dependency update."""
    TAG_LATEST = 35
    """States whether the release should be tagged with `latest`"""
    TAG_VERSION_SHORTCUTS = 36
    """States whether the release should be tagged with shortcuts i.e. major, major+minor"""
    SECRETS_BASELINE_FILENAME = 37
    """Filename for the detect-secrets baseline."""
    DOCUMENTATION_GUIDES_DIR = 38
    """Optional directory of Markdown guides to publish alongside generated API documentation."""
    DOCUMENTATION_GUIDES_OUTPUT_FOLDER = 39
    """Relative folder for rendered guides beneath the documentation output."""
    FAIL_ON_INCOMPLETE_LICENCE_AUDIT = 40
    """Fail the licence audit when dependencies or usable licence declarations are missing."""
    GENERATE_LICENSING_SUMMARY_ON_RELEASE = 41
    """Generate third-party licence summaries during a release when the plugin supports metadata."""
    LICENCE_ASSESSMENT_RULES_PATH = 42
    """Optional project TOML file overriding the built-in licence assessment policy."""
    LICENCE_ASSESSMENT_RULES = 43
    """Optional inline licence assessment rules in the project's pyproject.toml."""
    REVIEWED_LICENCE_ASSESSMENTS = 44
    """Project-specific records explaining manual reviews of REVIEW assessments."""
    LICENCE_ASSESSMENT_FAIL_ON = 45
    """Optional top-level list of assessment statuses that fail compliance checks."""
    SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING = 46
    """Skip automatic dependency downloads before licence and SPDX dependency analysis."""

    @staticmethod
    def choices() -> List[str]:
        """Gets a list of all possible configuration variables.

        Returns:
            a list of configuration variables
        """
        return [t.name.upper() for t in ConfigurationVariable]

    @staticmethod
    def parse(type_str: str) -> "ConfigurationVariable":
        """Determines the configuration variable from a string.

        Args:
            type_str: string to parse.

        Returns:
            corresponding configuration variable.
        """
        try:
            return ConfigurationVariable[type_str.upper()]
        except KeyError as e:
            raise ValueError(f"Unknown configuration variable: {type_str}. {e}")

Project's configuration variables.

Ancestors

• enum.Enum

Class variables

var ACCEPTED_THIRD_PARTY_LICENCES

List of accepted 3rd party licences for dependencies.

var AUTOGENERATE_NEWS_FILE_ON_DEPENDENCY_UPDATE

States whether news file should be generated on dependency updates.

var AWS_BUCKET

AWS bucket to use if needed.

var BETA_BRANCH

Name of beta/QA branch.

var BOT_EMAIL

Email of the Git bot account.

var BOT_USERNAME

Username of the Git bot account.

var CHANGELOG_FILE_PATH

Relative path to the changelog file.

var COPYRIGHT_START_DATE

Project's copyright start year.

var DEPENDENCY_UPDATE_BRANCH_PATTERN

Pattern of branches comprising dependency updates (e.g. dependabot).

var DEPENDENCY_UPDATE_NEWS_MESSAGE

Pattern of the news message for dependency update.

var DEPENDENCY_UPDATE_NEWS_TYPE

News file type for dependency update.

var DOCUMENTATION_DEFAULT_OUTPUT_PATH

The type of the None singleton.

var DOCUMENTATION_GUIDES_DIR

Optional directory of Markdown guides to publish alongside generated API documentation.

var DOCUMENTATION_GUIDES_OUTPUT_FOLDER

Relative folder for rendered guides beneath the documentation output.

var DOCUMENTATION_PRODUCTION_OUTPUT_PATH

Relative path to the folder where documentation will be generated.

var FAIL_ON_INCOMPLETE_LICENCE_AUDIT

Fail the licence audit when dependencies or usable licence declarations are missing.

var FILE_LICENCE_IDENTIFIER

SPDX identifier for the project's licence.

var GENERATE_LICENSING_SUMMARY_ON_RELEASE

Generate third-party licence summaries during a release when the plugin supports metadata.

var GIT_TOKEN

GIT token allowing committing back to the repository.

var IGNORE_REPOSITORY_TEST_UPLOAD

States whether to release package to a test registry (e.g. pypi test) test before release.

var LICENCE_ASSESSMENT_FAIL_ON

Optional top-level list of assessment statuses that fail compliance checks.

var LICENCE_ASSESSMENT_RULES

Optional inline licence assessment rules in the project's pyproject.toml.

var LICENCE_ASSESSMENT_RULES_PATH

Optional project TOML file overriding the built-in licence assessment policy.

var LOGGER_FORMAT

Tools' logger format.

var MASTER_BRANCH

Name of the main/production branch.

var MODULE_TO_DOCUMENT

Name of the module to generate documentation for.

var NEWS_DIR

Relative path to the directory comprising news files.

var ORGANISATION

Name of the organisation managing the project.

var ORGANISATION_EMAIL

Email of the organisation managing the project.

var PACKAGES_WITH_CHECKED_LICENCE

List of dependencies whose licence was manually checked.

var PACKAGE_NAME

Name of the package.

var PROGRAMMING_LANGUAGE

Project's programming language.

var PROJECT_CONFIG

Relative path to the file comprising project's configuration.

var PROJECT_NAME

Name of the project.

var PROJECT_ROOT

Relative path to the root of the project from pyproject.toml.

var PROJECT_UUID

UUID of the project to use in SPDX reports.

var RELEASE_BRANCH_PATTERN

Pattern of release branches.

var REMOTE_ALIAS

Git's remote alias.

var REVIEWED_LICENCE_ASSESSMENTS

Project-specific records explaining manual reviews of REVIEW assessments.

var SECRETS_BASELINE_FILENAME

Filename for the detect-secrets baseline.

var SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING

Skip automatic dependency downloads before licence and SPDX dependency analysis.

var SOURCE_DIR

Relative path to the source of the project.

var TAG_LATEST

States whether the release should be tagged with latest

var TAG_VERSION_SHORTCUTS

States whether the release should be tagged with shortcuts i.e. major, major+minor

var VERSION_FILE_PATH

Relative path to the file comprising project's version.

Static methods

def choices() ‑> List[str]

Expand source code

@staticmethod
def choices() -> List[str]:
    """Gets a list of all possible configuration variables.

    Returns:
        a list of configuration variables
    """
    return [t.name.upper() for t in ConfigurationVariable]

Gets a list of all possible configuration variables.

Returns -----= a list of configuration variables

def parse(type_str: str) ‑> ConfigurationVariable

Expand source code

@staticmethod
def parse(type_str: str) -> "ConfigurationVariable":
    """Determines the configuration variable from a string.

    Args:
        type_str: string to parse.

    Returns:
        corresponding configuration variable.
    """
    try:
        return ConfigurationVariable[type_str.upper()]
    except KeyError as e:
        raise ValueError(f"Unknown configuration variable: {type_str}. {e}")

Determines the configuration variable from a string.

Args

-----=

type_str

string to parse.

Returns -----= corresponding configuration variable.

class EnvironmentConfig

Expand source code

class EnvironmentConfig(GenericConfig):
    """Configuration set in environment variables.

    This also uses dotEnv mechanism.
    """

    def __init__(self) -> None:
        """Constructor."""
        dotenv.load_dotenv(dotenv.find_dotenv(usecwd=True, raise_error_if_not_found=False))

    def _fetch_value(self, key: str) -> Any:
        environment_value = os.getenv(key)
        if not environment_value:
            self._raise_undefined(key)
        return environment_value

Configuration set in environment variables.

This also uses dotEnv mechanism.

Constructor.

Ancestors

• GenericConfig
• abc.ABC

Inherited members

• GenericConfig:
  • get_value
  • get_value_or_default

class FileConfig (file_path: str | None = None)

Expand source code

class FileConfig(GenericConfig):
    """Configuration set in toml file.

    Note: any variable which relates to a PATH
    i.e. variable comprising one of the tokens in (PATH_TOKEN)
     will be modified and transformed in order to become absolute paths
     rather than relative paths as relative paths in the file are relative to
     the file location whereas relative paths when used by tools are relative .
     to current directory (i.e. os.getcwd()).
    """

    CONFIG_SECTION = "ProjectConfig"
    PATH_TOKEN = {"DIR", "ROOT", "PATH"}
    CONFIG_FILE_NAME = "pyproject.toml"

    def __init__(self, file_path: Optional[str] = None) -> None:
        """Constructor.

        Args:
            file_path: path to the toml configuration file.
        """
        self._file_path: Optional[str] = file_path
        self._config: Optional[dict] = None

    def _adjust_path_values(self, variable_name: str, value: str) -> str:
        """Works out the correct values for path variables.

        Paths in the configuration file are relative to the configuration
        file location. This method ensures the path values are therefore
        evaluated properly
        Args:
            variable_name: name of the variable in the configuration file
            value: variable value

        Returns:
            a valid path or the value unchanged if the variable is not a path

        """
        if not self._file_path:
            return value
        for token in FileConfig.PATH_TOKEN:
            if token in variable_name:
                config_file_dir = os.path.dirname(self._file_path)
                resolved_path = os.path.join(config_file_dir, value)
                value = os.path.realpath(resolved_path)
                break
        return value

    @staticmethod
    def _look_for_config_file_walking_up_tree() -> Optional[str]:
        try:
            return str(find_file_in_tree(FileConfig.CONFIG_FILE_NAME, top=True))
        except FileNotFoundError as e:
            logger.warning(e)
        return None

    @staticmethod
    def _find_config_file(file_path: Optional[str]) -> Optional[str]:
        if file_path and os.path.exists(file_path):
            return file_path
        try:
            return str(find_file_in_tree(FileConfig.CONFIG_FILE_NAME))
        except FileNotFoundError:
            return FileConfig._look_for_config_file_walking_up_tree()

    @staticmethod
    def _load_config_from_file(file_path: str) -> Dict[str, Any]:
        config: dict = toml.load(file_path).get(FileConfig.CONFIG_SECTION, dict())
        config[ConfigurationVariable.PROJECT_CONFIG.name] = file_path
        return config

    @property
    def config(self) -> dict:
        """Gets the file configuration."""
        if not self._config:
            self._file_path = FileConfig._find_config_file(self._file_path)
            self._config = FileConfig._load_config_from_file(self._file_path) if self._file_path else dict()
        return self._config

    def _fetch_value(self, key: str) -> Any:
        try:
            return self._adjust_path_values(key, self.config[key])
        except KeyError:
            self._raise_undefined(key)

Configuration set in toml file.

Note: any variable which relates to a PATH i.e. variable comprising one of the tokens in (PATH_TOKEN) will be modified and transformed in order to become absolute paths rather than relative paths as relative paths in the file are relative to the file location whereas relative paths when used by tools are relative . to current directory (i.e. os.getcwd()).

Constructor.

Args

-----=

file_path

path to the toml configuration file.

Ancestors

• GenericConfig
• abc.ABC

Class variables

var CONFIG_FILE_NAME

The type of the None singleton.

var CONFIG_SECTION

The type of the None singleton.

var PATH_TOKEN

The type of the None singleton.

Instance variables

prop config : dict

Expand source code

@property
def config(self) -> dict:
    """Gets the file configuration."""
    if not self._config:
        self._file_path = FileConfig._find_config_file(self._file_path)
        self._config = FileConfig._load_config_from_file(self._file_path) if self._file_path else dict()
    return self._config

Gets the file configuration.

Inherited members

• GenericConfig:
  • get_value
  • get_value_or_default

class GenericConfig

Expand source code

class GenericConfig(ABC):
    """Abstract Class for determining configuration values."""

    @abstractmethod
    def _fetch_value(self, key: str) -> Any:
        self._raise_undefined(key)

    def _raise_undefined(self, key: Optional[str]) -> None:
        raise Undefined(f"Undefined key: {key}")

    def get_value(self, key: Union[str, ConfigurationVariable]) -> Any:
        """Gets a configuration value.

        If the variable was not defined, an exception is raised.

        Args:
            key: variable key. This can be a string or a ConfigurationVariable
            element.

        Returns:
            configuration value corresponding to the key.
        """
        if not key:
            raise KeyError(key)
        key_str = key.name if isinstance(key, ConfigurationVariable) else key
        return self._fetch_value(key_str)

    def get_value_or_default(self, key: Union[str, ConfigurationVariable], default_value: Any) -> Any:
        """Gets a configuration value.

        If the variable was not defined, the default value is returned.

        Args:
            key: variable key. This can be a string or a ConfigurationVariable
            element.
            default_value: value to default to if the variable was not defined.

        Returns:
            configuration value corresponding to the key.
            default value if the variable is not defined.
        """
        try:
            return self.get_value(key)
        except Undefined as e:
            logger.debug(e)
            return default_value

Abstract Class for determining configuration values.

Ancestors

• abc.ABC

Subclasses

• EnvironmentConfig
• FileConfig
• ProjectConfiguration
• StaticConfig

Methods

def get_value(self,
key: str | ConfigurationVariable) ‑> Any

Expand source code

def get_value(self, key: Union[str, ConfigurationVariable]) -> Any:
    """Gets a configuration value.

    If the variable was not defined, an exception is raised.

    Args:
        key: variable key. This can be a string or a ConfigurationVariable
        element.

    Returns:
        configuration value corresponding to the key.
    """
    if not key:
        raise KeyError(key)
    key_str = key.name if isinstance(key, ConfigurationVariable) else key
    return self._fetch_value(key_str)

Gets a configuration value.

If the variable was not defined, an exception is raised.

Args

-----=

key

variable key. This can be a string or a ConfigurationVariable

element.

Returns -----= configuration value corresponding to the key.

def get_value_or_default(self,
key: str | ConfigurationVariable,
default_value: Any) ‑> Any

Expand source code

def get_value_or_default(self, key: Union[str, ConfigurationVariable], default_value: Any) -> Any:
    """Gets a configuration value.

    If the variable was not defined, the default value is returned.

    Args:
        key: variable key. This can be a string or a ConfigurationVariable
        element.
        default_value: value to default to if the variable was not defined.

    Returns:
        configuration value corresponding to the key.
        default value if the variable is not defined.
    """
    try:
        return self.get_value(key)
    except Undefined as e:
        logger.debug(e)
        return default_value

Gets a configuration value.

If the variable was not defined, the default value is returned.

Args

-----=

key

variable key. This can be a string or a ConfigurationVariable

element.

default_value

value to default to if the variable was not defined.

Returns -----= configuration value corresponding to the key. default value if the variable is not defined.

class ProjectConfiguration (sources: List[GenericConfig])

Expand source code

class ProjectConfiguration(GenericConfig):
    """Overall project's configuration."""

    def __init__(self, sources: List[GenericConfig]):
        """Constructor.

        Args:
            sources: list of configuration sources
        """
        self._config_sources: list = sources

    def _fetch_value(self, key: str) -> Any:
        for config in self._config_sources:
            try:
                return config.get_value(key)
            except Undefined:
                pass
        else:
            self._raise_undefined(key)

Overall project's configuration.

Constructor.

Args

-----=

sources

list of configuration sources

Ancestors

• GenericConfig
• abc.ABC

Inherited members

• GenericConfig:
  • get_value
  • get_value_or_default

class StaticConfig

Expand source code

class StaticConfig(GenericConfig):
    """Configuration with default values.

    Only variables which are not likely do be different from a project to
    another are defined here. They can be overridden by values in the
    configuration file though. This should simply the number of variables
    defined in toml.
    """

    BETA_BRANCH = "beta"
    MASTER_BRANCH = "master"
    RELEASE_BRANCH_PATTERN = r"^release.*$"
    REMOTE_ALIAS = "origin"
    LOGGER_FORMAT = "%(levelname)s: %(message)s"
    BOT_USERNAME = "Monty Bot"
    BOT_EMAIL = "monty-bot@arm.com"
    ORGANISATION = "Arm Limited"
    ORGANISATION_EMAIL = "support@arm.com"
    FILE_LICENCE_IDENTIFIER = "Apache-2.0"
    COPYRIGHT_START_DATE = 2020
    PROGRAMMING_LANGUAGE = "NoOp"
    AWS_BUCKET = "Unknown"
    AUTOGENERATE_NEWS_FILE_ON_DEPENDENCY_UPDATE = True
    TAG_LATEST = False
    TAG_VERSION_SHORTCUTS = False
    SECRETS_BASELINE_FILENAME = ".secrets.baseline"
    FAIL_ON_INCOMPLETE_LICENCE_AUDIT = False
    LICENCE_ASSESSMENT_FAIL_ON = None
    """When unset, use fail_on from the embedded or project licence assessment rules."""
    GENERATE_LICENSING_SUMMARY_ON_RELEASE = False
    SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING = False
    DEPENDENCY_UPDATE_NEWS_MESSAGE = "Dependency upgrade: {message}"
    DEPENDENCY_UPDATE_NEWS_TYPE = NewsType.bugfix
    DEPENDENCY_UPDATE_BRANCH_PATTERN = r"^\s*[Dd]ependabot\/.+\/(?P<DEPENDENCY>.+)"
    ACCEPTED_THIRD_PARTY_LICENCES = [
        "Apache-2.0",
        "BSD*",
        "CC-BY-*",
        "JSON",
        "MIT",
        "Python-2.0",
        "PSF-2.0",
        "MPL-2.0",
    ]
    PACKAGES_WITH_CHECKED_LICENCE: List[str] = []

    def _fetch_value(self, key: str) -> Any:
        try:
            return getattr(self, key)
        except AttributeError:
            self._raise_undefined(key)

Configuration with default values.

Only variables which are not likely do be different from a project to another are defined here. They can be overridden by values in the configuration file though. This should simply the number of variables defined in toml.

Ancestors

• GenericConfig
• abc.ABC

Class variables

var ACCEPTED_THIRD_PARTY_LICENCES

The type of the None singleton.

var AUTOGENERATE_NEWS_FILE_ON_DEPENDENCY_UPDATE

The type of the None singleton.

var AWS_BUCKET

The type of the None singleton.

var BETA_BRANCH

The type of the None singleton.

var BOT_EMAIL

The type of the None singleton.

var BOT_USERNAME

The type of the None singleton.

var COPYRIGHT_START_DATE

The type of the None singleton.

var DEPENDENCY_UPDATE_BRANCH_PATTERN

The type of the None singleton.

var DEPENDENCY_UPDATE_NEWS_MESSAGE

The type of the None singleton.

var DEPENDENCY_UPDATE_NEWS_TYPE

The type of the None singleton.

var FAIL_ON_INCOMPLETE_LICENCE_AUDIT

The type of the None singleton.

var FILE_LICENCE_IDENTIFIER

The type of the None singleton.

var GENERATE_LICENSING_SUMMARY_ON_RELEASE

The type of the None singleton.

var LICENCE_ASSESSMENT_FAIL_ON

When unset, use fail_on from the embedded or project licence assessment rules.

var LOGGER_FORMAT

The type of the None singleton.

var MASTER_BRANCH

The type of the None singleton.

var ORGANISATION

The type of the None singleton.

var ORGANISATION_EMAIL

The type of the None singleton.

var PACKAGES_WITH_CHECKED_LICENCE : List[str]

The type of the None singleton.

var PROGRAMMING_LANGUAGE

The type of the None singleton.

var RELEASE_BRANCH_PATTERN

The type of the None singleton.

var REMOTE_ALIAS

The type of the None singleton.

var SECRETS_BASELINE_FILENAME

The type of the None singleton.

var SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING

The type of the None singleton.

var TAG_LATEST

The type of the None singleton.

var TAG_VERSION_SHORTCUTS

The type of the None singleton.

Inherited members

• GenericConfig:
  • get_value
  • get_value_or_default

class Undefined (*args, **kwargs)

Expand source code

class Undefined(Exception):
    """Exception raised when a configuration value is not defined."""

    pass

Exception raised when a configuration value is not defined.

Ancestors

• builtins.Exception
• builtins.BaseException