Other Modules

curator.cli

curator.cli.process_action(client, action_def, dry_run=False)

Do the action in action_def.action, using the associated options and any kwargs.

Parameters:
Return type:

None
curator.cli.override_logging(config, loglevel, logfile, logformat)

Get logging config and override from command-line options

Parameters:
  • config (dict) – The configuration from file
  • loglevel (str) – The log level
  • logfile (str) – The log file to write
  • logformat (str) – Which log format to use
Returns:

Log configuration ready for validation
Return type:

dict
curator.cli.cli_hostslist(hosts)
Parameters:hosts (str or list) – One or more hosts.
Returns:A list of hosts that came in from the command-line, or None
Return type:list or None
curator.cli.ilm_action_skip(client, action_def)

Skip rollover action if allow_ilm_indices is false. For all other non-snapshot actions, add the ilm filtertype to the filters list.

Parameters:action_def (ActionDef) – An action object
Returns:True if action_def.action is rollover and the alias identified by action_def.options['name'] is associated with an ILM policy. This hacky work-around is because the Rollover action does not use IndexList
Return type:bool
curator.cli.exception_handler(action_def, err)

Do the grunt work with the exception

Parameters:
curator.cli.run(client_args, other_args, action_file, dry_run=False)

Called by cli() to execute what was collected at the command-line

Parameters:
  • client_args (ClientArgs) – The ClientArgs arguments object
  • other_args (OtherArgs) – The OtherArgs arguments object
  • action_file (str) – The action configuration file
  • dry_run (bool) – Do not perform any changes
curator.cli.cli(ctx, config, hosts, cloud_id, id, api_key, username, password, bearer_auth, opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key, ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test, dry_run, loglevel, logfile, logformat, action_file)

This is the click.Command that initiates everything and connects the command-line to the rest of Curator.

Parameters:
  • ctx (Context) – The Click Context
  • config (str) – Path to configuration file.
  • hosts (list) – Elasticsearch URL to connect to
  • cloud_id (str) – Shorthand to connect to Elastic Cloud instance
  • id (str) – API Key “id” value
  • api_key (str) – API Key “api_key” value
  • username (str) – Username used to create “basic_auth” tuple
  • password (str) – Password used to create “basic_auth” tuple
  • bearer_auth (str) – Bearer Auth Token
  • opaque_id (str) – Opaque ID string
  • request_timeout (int) – Request timeout in seconds
  • http_compress (bool) – Enable HTTP compression
  • verify_certs (bool) – Verify SSL/TLS certificate(s)
  • ca_certs (str) – Path to CA certificate file or directory
  • client_cert (str) – Path to client certificate file
  • client_key (str) – Path to client certificate key
  • ssl_assert_hostname (str) – Hostname or IP address to verify on the node’s certificate.
  • ssl_assert_fingerprint (str) – SHA-256 fingerprint of the node’s certificate. If this value is given then root-of-trust verification isn’t done and only the node’s certificate fingerprint is verified.
  • ssl_version (str) – Minimum acceptable TLS/SSL version
  • master_only (bool) – Only run if the single host provided is the elected master
  • skip_version_test (bool) – Do not check the host version
  • dry_run (bool) – Do not perform any changes.
  • loglevel (str) – Log level
  • logfile (str) – Path to log file
  • logformat (str) – Log output format
  • action_file (str) – Path to action file

curator.config_utils

curator.config_utils.check_logging_config(config)

Ensure that the top-level key logging is in config before passing it to SchemaCheck for value validation.

Parameters:config (dict) – Logging configuration data
Returns:SchemaCheck validated logging configuration.
curator.config_utils.set_logging(log_opts)

Configure global logging options

Parameters:log_opts (dict) – Logging configuration data
Return type:None
curator.config_utils.password_filter(data)

Recursively look through all nested structures of data for the key 'password' and redact the value.

Parameters:data (dict) – Configuration data
Returns:A deepcopy of data with the value obscured by REDACTED if the key is 'password'.

curator.logtools

curator.logtools.de_dot(dot_string, msg)

Turn message and dotted string into a nested dictionary. Used by LogstashFormatter

Parameters:
  • dot_string (str) – The dotted string
  • msg (str) – The message
curator.logtools.deepmerge(source, destination)

Recursively merge deeply nested dictionary structures, source into destination

Parameters:
  • source (dict) – Source dictionary
  • destination (dict) – Destination dictionary
Returns:

destination
Return type:

dict
class curator.logtools.LogInfo(cfg)

Logging Class

Class Setup

Parameters:cfg – The logging configuration
Type:cfg: dict

curator.logtools.LogstashFormatter

This inherits from logging.Formatter, so some of what you see documented is inherited.

class curator.logtools.LogstashFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Logstash formatting (JSON)

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

Changed in version 3.2: Added the style parameter.

curator.repomgrcli

curator.repomgrcli.delete_callback(ctx, param, value)

Callback if command delete called

If the action is delete, this is the click.Parameter callback function if you used the --yes flag.

Parameters:
  • ctx (Context) – The Click Context
  • param (str) – The parameter name
  • value (str) – The value
curator.repomgrcli.show_repos(client)

Show all repositories

Parameters:client (Elasticsearch) – A client connection object
Return type:None
curator.repomgrcli.get_client(ctx)
Parameters:ctx (Context) – The click Context
Returns:A client connection object
Return type:Elasticsearch
curator.repomgrcli.create_repo(ctx, repo_name=None, repo_type=None, repo_settings=None, verify=False)

Call create_repository() to create a snapshot repository from the provided arguments

Parameters:
  • ctx (Context) – The click Context
  • repo_name (str) – The repository name
  • repo_type (str) – The repository name
  • repo_settings (dict) – Settings to configure the repository
  • verify (bool) – Whether to verify repository access
Return type:

None
curator.repomgrcli.azure(ctx, name, client, container, base_path, chunk_size, compress, max_restore_rate, max_snapshot_rate, readonly, location_mode, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • client (str) – The named client (Azure)
  • container (str) – Container name. You must create the Azure container before creating the repository.
  • base_path (str) – Specifies the path within container to repository data. Defaults to empty (root directory).
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • compress (bool) – Enable/Disable metadata compression.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • max_snapshot_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • readonly (bool) – Make repsitory read-only.
  • location_mode (str) – Either primary_only or secondary_only. Note that if you set it to secondary_only, it will force readonly to True.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.gcs(ctx, name, bucket, client, base_path, chunk_size, compress, max_restore_rate, max_snapshot_rate, readonly, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • client (str) – The name of the client to use to connect to Google Cloud Storage.
  • bucket (str) – The name of the bucket to be used for snapshots.
  • base_path (str) – Specifies the path within bucket to repository data. Defaults to the root of the bucket.
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • compress (bool) – Enable/Disable metadata compression.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • max_snapshot_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • readonly (bool) – Make repsitory read-only.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.s3(ctx, name, bucket, client, base_path, chunk_size, compress, max_restore_rate, max_snapshot_rate, readonly, server_side_encryption, buffer_size, canned_acl, storage_class, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • bucket (str) – The bucket name must adhere to Amazon’s S3 bucket naming rules.
  • client (str) – The name of the S3 client to use to connect to S3.
  • base_path (str) – Specifies the path to the repository data within its bucket. Defaults to an empty string, meaning that the repository is at the root of the bucket. The value of this setting should not start or end with a /.
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • compress (bool) – Enable/Disable metadata compression.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • max_snapshot_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • readonly (bool) – Make repsitory read-only.
  • server_side_encryption (bool) – If set, files are encrypted on server side using AES256 algorithm.
  • buffer_size (str) – Minimum threshold below which the chunk is uploaded using a single request. Must be between 5mb and 5gb.
  • canned_acl (str) – When the S3 repository creates buckets and objects, it adds the canned ACL into the buckets and objects.
  • storage_class (str) – Sets the S3 storage class for objects stored in the snapshot repository.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.fs(ctx, name, location, compress, chunk_size, max_snapshots, max_restore_rate, max_snapshot_rate, readonly, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • location (str) – Shared file-system location. Must match remote path, & be accessible to all master & data nodes
  • compress (bool) – Enable/Disable metadata compression.
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • max_snapshots (int) – Maximum number of snapshots the repository can contain. Defaults to Integer.MAX_VALUE, which is 2147483647.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • max_snapshot_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • readonly (bool) – Make repsitory read-only.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.url(ctx, name, chunk_size, http_max_retries, http_socket_timeout, compress, max_snapshots, max_restore_rate, shared_filesystem_url, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • http_max_retries (int) – Maximum number of retries for http and https
  • http_socket_timeout (int) – Maximum wait time for data transfers over a connection.
  • compress (bool) – Enable/Disable metadata compression.
  • max_snapshots (int) – Maximum number of snapshots the repository can contain. Defaults to Integer.MAX_VALUE, which is 2147483647.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • shared_filesystem_url (str) – URL location of the root of the shared filesystem repository.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.source(ctx, name, delegate_type, location, compress, chunk_size, max_snapshots, max_restore_rate, max_snapshot_rate, readonly, verify)
Parameters:
  • ctx (Context) – The Click Context
  • name (str) – The repository name
  • delegate_type (str) – Delegated repository type.
  • location (str) – Shared file-system location. Must match remote path, & be accessible to all master & data nodes
  • compress (bool) – Enable/Disable metadata compression.
  • chunk_size (str) – Chunk size, e.g. 1g, 10m, 5k. Default is unbounded.
  • max_snapshots (int) – Maximum number of snapshots the repository can contain. Defaults to Integer.MAX_VALUE, which is 2147483647.
  • max_restore_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • max_snapshot_rate (str) – Throttles per node restore rate (per second). Default is 20mb.
  • readonly (bool) – Make repsitory read-only.
  • verify (bool) – Verify repository after creation.
curator.repomgrcli.repo_mgr_cli(ctx, config, hosts, cloud_id, id, api_key, username, password, bearer_auth, opaque_id, request_timeout, http_compress, verify_certs, ca_certs, client_cert, client_key, ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, master_only, skip_version_test, dry_run, loglevel, logfile, logformat)
Parameters:
  • ctx (Context) – The Click Context
  • config (str) – Path to configuration file.
  • hosts (list) – Elasticsearch URL to connect to
  • cloud_id (str) – Shorthand to connect to Elastic Cloud instance
  • id (str) – API Key “id” value
  • api_key (str) – API Key “api_key” value
  • username (str) – Username used to create “basic_auth” tuple
  • password (str) – Password used to create “basic_auth” tuple
  • bearer_auth (str) – Bearer Auth Token
  • opaque_id (str) – Opaque ID string
  • request_timeout (int) – Request timeout in seconds
  • http_compress (bool) – Enable HTTP compression
  • verify_certs (bool) – Verify SSL/TLS certificate(s)
  • ca_certs (str) – Path to CA certificate file or directory
  • client_cert (str) – Path to client certificate file
  • client_key (str) – Path to client certificate key
  • ssl_assert_hostname (str) – Hostname or IP address to verify on the node’s certificate.
  • ssl_assert_fingerprint (str) – SHA-256 fingerprint of the node’s certificate. If this value is given then root-of-trust verification isn’t done and only the node’s certificate fingerprint is verified.
  • ssl_version (str) – Minimum acceptable TLS/SSL version
  • master_only (bool) – Only run if the single host provided is the elected master
  • skip_version_test (bool) – Do not check the host version
  • dry_run (bool) – Do not perform any changes. NON-FUNCTIONAL PLACEHOLDER! DO NOT USE!
  • loglevel (str) – Log level
  • logfile (str) – Path to log file
  • logformat (str) – Log output format