Helpers¶
Date Ops¶
-
class
curator.helpers.date_ops.
TimestringSearch
(timestring)¶ Bases:
object
An object to allow repetitive search against a string,
searchme
, without having to repeatedly recreate the regex.Parameters: timestring ( strftime()
) – Anstrftime
pattern-
get_epoch
(searchme)¶ Parameters: searchme – A string to be matched against pattern
that matchestimestring
Returns: The epoch timestamp extracted from searchme
by regex matching againstpattern
Return type: int
-
pattern
= None¶ Object attribute.
re.compile(regex)
whereregex = r'(?P<date>{0})'.format(get_date_regex(timestring))
. Usesget_date_regex()
-
timestring
= None¶ Object attribute preserving param
timestring
-
-
curator.helpers.date_ops.
absolute_date_range
(unit, date_from, date_to, date_from_format=None, date_to_format=None)¶ This function calculates a date range with an absolute time stamp for both the start time and the end time. These dates are converted to epoch time. The parameter
unit
is used when the same simplified date is used for bothdate_from
anddate_to
to calculate the duration. For example, ifunit
ismonths
, anddate_from
anddate_to
are both2017.01
, then the entire month of January 2017 will be the absolute date range.Parameters: - unit (str) – One of
hours
,days
,weeks
,months
, oryears
. - date_from (str) – The simplified date for the start of the range
- date_to (str) – The simplified date for the end of the range.
- date_from_format (str) – The
strftime()
string used to parsedate_from
- date_to_format (str) – The
strftime()
string used to parsedate_to
Returns: The epoch start time and end time of a date range
Return type: - unit (str) – One of
-
curator.helpers.date_ops.
date_range
(unit, range_from, range_to, epoch=None, week_starts_on='sunday')¶ This function calculates a date range with a distinct epoch time stamp of both the start time and the end time in counts of
unit
relative to the time at execution, ifepoch
is not set.If
unit
isweeks
, you can also determine when a week begins usingweek_starts_on
, which can be eithersunday
ormonday
.Parameters: - unit (str) – One of
hours
,days
,weeks
,months
, oryears
. - range_from (int) – Count of
unit
in the past/future is the origin? - range_to (int) – Count of
unit
in the past/future is the end point? - epoch (int) – An epoch timestamp used to establish a point of reference for calculations.
- week_starts_on (str) – Either
sunday
ormonday
. Default issunday
Returns: The epoch start time and end time of a date range
Return type: - unit (str) – One of
-
curator.helpers.date_ops.
datetime_to_epoch
(mydate)¶ Converts datetime into epoch seconds
Parameters: mydate ( datetime
) – A Python datetimeReturns: An epoch timestamp based on mydate
Return type: int
-
curator.helpers.date_ops.
fix_epoch
(epoch)¶ Fix value of
epoch
to be the count since the epoch in seconds only, which should be 10 or fewer digits long.Parameters: epoch (int) – An epoch timestamp, in epoch + milliseconds, or microsecond, or even nanoseconds. Returns: An epoch timestamp in seconds only, based on epoch
Return type: int
-
curator.helpers.date_ops.
get_date_regex
(timestring)¶ Parameters: timestring ( strftime()
) – Anstrftime
patternReturns: A regex string based on a provided strftime()
timestring
.Return type: str
-
curator.helpers.date_ops.
get_datemath
(client, datemath, random_element=None)¶ Parameters: - client (
Elasticsearch
) – A client connection object - datemath (
datemath
) – An elasticsearch datemath string - random_element (str) – This allows external randomization of the name and is only useful for tests so that you can guarantee the output because you provided the random portion.
Returns: the parsed index name from
datemath
Return type: - client (
-
curator.helpers.date_ops.
get_datetime
(index_timestamp, timestring)¶ Parameters: - index_timestamp (str) – The index timestamp
- timestring (
strftime()
) – Anstrftime
pattern
Returns: The datetime extracted from the index name, which is the index creation time.
Return type:
-
curator.helpers.date_ops.
get_point_of_reference
(unit, count, epoch=None)¶ Parameters: - unit (str) – One of
seconds
,minutes
,hours
,days
,weeks
,months
, oryears
. - unit_count (int) – The number of
units
.unit_count
*unit
will be calculated out to the relative number of seconds. - epoch (int) – An epoch timestamp used in conjunction with
unit
andunit_count
to establish a point of reference for calculations.
Returns: A point-of-reference timestamp in epoch + milliseconds by deriving from a
unit
and acount
, and an optional reference timestamp,epoch
Return type: - unit (str) – One of
-
curator.helpers.date_ops.
get_unit_count_from_name
(index_name, pattern)¶ Parameters: Returns: The unit count, if a match is able to be found in the name
Return type:
-
curator.helpers.date_ops.
handle_iso_week_number
(mydate, timestring, index_timestamp)¶ Parameters: - mydate (
datetime
) – A Python datetime - timestring (
strftime()
) – Anstrftime
pattern - index_timestamp (str) – The index timestamp
Returns: The date of the previous week based on ISO week number
Return type: - mydate (
-
curator.helpers.date_ops.
isdatemath
(data)¶ Parameters: data (str) – An expression to validate as being datemath or not Returns: True
ifdata
is a valid datemath expression, elseFalse
Return type: bool
-
curator.helpers.date_ops.
parse_date_pattern
(name)¶ Scan and parse
name
forstrftime()
strings, replacing them with the associated value when found, but otherwise returning lowercase values, as uppercase snapshot names are not allowed. It will detect if the first character is a<
, which would indicatename
is going to be using Elasticsearch date math syntax, and skip accordingly.The
strftime()
identifiers that Curator currently recognizes as acceptable include:Y
: A 4 digit yeary
: A 2 digit yearm
: The 2 digit monthW
: The 2 digit week of the yeard
: The 2 digit day of the monthH
: The 2 digit hour of the day, in 24 hour notationM
: The 2 digit minute of the hourS
: The 2 digit number of second of the minutej
: The 3 digit day of the year
Parameters: name (str) – A name, which can contain strftime()
stringsReturns: The parsed date pattern Return type: str
-
curator.helpers.date_ops.
parse_datemath
(client, value)¶ Validate that
value
looks like proper datemath. If it passes this test, then try to ship it to Elasticsearch for real. It may yet fail this test, and if it does, it will raise aConfigurationError
exception. If it passes, return the fully parsed string.Parameters: - client (
Elasticsearch
) – A client connection object - value (str) – A string to check for datemath
Returns: A datemath indexname, fully rendered by Elasticsearch
Return type: - client (
Getters¶
-
curator.helpers.getters.
byte_size
(num, suffix='B')¶ Parameters: Returns: A formatted string indicating the size in bytes, with the proper unit, e.g. KB, MB, GB, TB, etc.
Return type:
-
curator.helpers.getters.
get_client
(configdict=None, configfile=None, autoconnect=False, version_min=(8, 0, 0), version_max=(8, 99, 99))¶ Get an Elasticsearch Client using
es_client.Builder
Build a client out of settings from configfile or configdict If neither configfile nor configdict is provided, empty defaults will be used. If both are provided, configdict will be used, and configfile ignored.
Parameters: Returns: A client connection object
Return type:
-
curator.helpers.getters.
get_data_tiers
(client)¶ Get all valid data tiers from the node roles of each node in the cluster by polling each node
Parameters: client ( Elasticsearch
) – A client connection objectReturns: The available data tiers in tier: bool
form.Return type: dict
-
curator.helpers.getters.
get_indices
(client)¶ Calls
get_settings()
Parameters: client ( Elasticsearch
) – A client connection objectReturns: The current list of indices from the cluster Return type: list
-
curator.helpers.getters.
get_repository
(client, repository='')¶ Calls
get_repository()
Parameters: - client (
Elasticsearch
) – A client connection object - repository (str) – The Elasticsearch snapshot repository to use
Returns: Configuration information for
repository
.Return type: - client (
-
curator.helpers.getters.
get_snapshot
(client, repository=None, snapshot='')¶ Calls
get()
Parameters: - client (
Elasticsearch
) – A client connection object - repository (str) – The Elasticsearch snapshot repository to use
- snapshot (str) – The snapshot name, or a comma-separated list of snapshots
Returns: Information about the provided
snapshot
, a snapshot (or a comma-separated list of snapshots). If no snapshot specified, it will collect info for all snapshots. If none exist, an emptydict
will be returned.Return type: - client (
-
curator.helpers.getters.
get_snapshot_data
(client, repository=None)¶ Get all snapshots from repository and return a list. Calls
get()
Parameters: - client (
Elasticsearch
) – A client connection object - repository (str) – The Elasticsearch snapshot repository to use
Returns: The list of all snapshots from
repository
Return type: - client (
-
curator.helpers.getters.
get_write_index
(client, alias)¶ Calls
get_alias()
Parameters: - client (
Elasticsearch
) – A client connection object - alias (str) – An alias name
Returns: The the index name associated with the alias that is designated
is_write_index
Return type: - client (
-
curator.helpers.getters.
index_size
(client, idx, value='total')¶ Calls
stats()
Parameters: - client (
Elasticsearch
) – A client connection object - idx (str) – An index name
- value (str) – One of either
primaries
ortotal
Returns: The sum of either
primaries
ortotal
shards for indexidx
Return type: integer
- client (
-
curator.helpers.getters.
name_to_node_id
(client, name)¶ Calls
stats()
Parameters: - client (
Elasticsearch
) – A client connection object - name (str) – The node
name
Returns: The node_id of the node identified by
name
Return type: - client (
-
curator.helpers.getters.
node_id_to_name
(client, node_id)¶ Calls
stats()
Parameters: - client (
Elasticsearch
) – A client connection object - node_id (str) – The node
node_id
Returns: The name of the node identified by
node_id
Return type: - client (
-
curator.helpers.getters.
node_roles
(client, node_id)¶ Calls
info()
Parameters: - client (
Elasticsearch
) – A client connection object - node_id (str) – The node
node_id
Returns: The list of roles assigned to the node identified by
node_id
Return type: - client (
-
curator.helpers.getters.
single_data_path
(client, node_id)¶ In order for a shrink to work, it should be on a single filesystem, as shards cannot span filesystems. Calls
stats()
Parameters: - client (
Elasticsearch
) – A client connection object - node_id (str) – The node
node_id
Returns: True
if the node has a single filesystem, elseFalse
Return type: - client (
Testers¶
-
curator.helpers.testers.
ilm_policy_check
(client, alias)¶ Test if alias is associated with an ILM policy
Calls
get_settings()
Parameters: - client (
Elasticsearch
) – A client connection object - alias (str) – The alias name
Return type: - client (
-
curator.helpers.testers.
repository_exists
(client, repository=None)¶ Calls
get_repository()
Parameters: - client (
Elasticsearch
) – A client connection object - repository (str) – The Elasticsearch snapshot repository to use
Returns: True
ifrepository
exists, elseFalse
Return type: - client (
-
curator.helpers.testers.
rollable_alias
(client, alias)¶ Calls
get_alias()
Parameters: - client (
Elasticsearch
) – A client connection object - alias (str) – An Elasticsearch alias
Returns: True
orFalse
depending on whetheralias
is an alias that points to an index that can be used by the_rollover
API.Return type: - client (
-
curator.helpers.testers.
snapshot_running
(client)¶ Calls
get_repository()
Return
True
if a snapshot is in progress, andFalse
if notParameters: client ( Elasticsearch
) – A client connection objectReturn type: bool
-
curator.helpers.testers.
validate_actions
(data)¶ Validate the
actions
configuration dictionary, as imported from actions.yml, for example.Parameters: data (dict) – The configuration dictionary Returns: The validated and sanitized configuration dictionary. Return type: dict
-
curator.helpers.testers.
validate_filters
(action, myfilters)¶ Validate that myfilters are appropriate for the action type, e.g. no index filters applied to a snapshot list.
Parameters: Returns: Validated list of filters
Return type:
-
curator.helpers.testers.
verify_client_object
(test)¶ Parameters: test ( Elasticsearch
) – The variable or object to testReturns: True
iftest
is a properElasticsearch
client object and raise aTypeError
exception if it is not.Return type: bool
-
curator.helpers.testers.
verify_repository
(client, repository=None)¶ Do
verify_repository()
call. If it fails, raise aRepositoryException
.Parameters: - client (
Elasticsearch
) – A client connection object - repository (str) – A repository name
Return type: - client (
-
curator.helpers.testers.
verify_snapshot_list
(test)¶ Parameters: test ( SnapshotList
) – The variable or object to testReturns: None
iftest
is a properSnapshotList
object, else raise aTypeError
exception.Return type: None
Utils¶
-
curator.helpers.utils.
chunk_index_list
(indices)¶ This utility chunks very large index lists into 3KB chunks. It measures the size as a csv string, then converts back into a list for the return value.
Parameters: indices (list) – The list of indices Returns: A list of lists (each a piece of the original indices
)Return type: list
-
curator.helpers.utils.
report_failure
(exception)¶ Raise a
FailedExecution
exception and include the original error message.Parameters: exception (:py:exc:Exception) – The upstream exception. Return type: None
-
curator.helpers.utils.
show_dry_run
(ilo, action, **kwargs)¶ Log dry run output with the action which would have been executed.
Parameters: Return type:
Waiters¶
-
curator.helpers.waiters.
health_check
(client, **kwargs)¶ This function calls client.cluster.
health()
and, based on the params provided, will returnTrue
orFalse
depending on whether that particular keyword appears in the output, and has the expected value.If multiple keys are provided, all must match for a
True
response.Parameters: client ( Elasticsearch
) – A client connection objectReturn type: bool
-
curator.helpers.waiters.
relocate_check
(client, index)¶ This function calls client.cluster.
state()
with a given index to check if all of the shards for that index are in theSTARTED
state. It will returnTrue
if all primary and replica shards are in theSTARTED
state, and it will returnFalse
if any shard is in a different state.Parameters: - client (
Elasticsearch
) – A client connection object - index (str) – The index name
Return type: - client (
-
curator.helpers.waiters.
restore_check
(client, index_list)¶ This function calls client.indices.
recovery()
with the list of indices to check for complete recovery. It will returnTrue
if recovery of those indices is complete, andFalse
otherwise. It is designed to fail fast: if a single shard is encountered that is still recovering (not inDONE
stage), it will immediately returnFalse
, rather than complete iterating over the rest of the response.Parameters: - client (
Elasticsearch
) – A client connection object - index_list (list) – The list of indices to verify having been restored.
Return type: - client (
-
curator.helpers.waiters.
snapshot_check
(client, snapshot=None, repository=None)¶ This function calls client.snapshot.
get()
and tests to see whether the snapshot is complete, and if so, with what status. It will log errors according to the result. If the snapshot is stillIN_PROGRESS
, it will returnFalse
.SUCCESS
will be anINFO
level message,PARTIAL
nets aWARNING
message,FAILED
is anERROR
, message, and all others will be aWARNING
level message.Parameters: - client (
Elasticsearch
) – A client connection object - snapshot (str) – The snapshot name
- repository (str) – The repository name
Return type: - client (
-
curator.helpers.waiters.
task_check
(client, task_id=None)¶ This function calls client.tasks.
get()
with the providedtask_id
. If the task data contains'completed': True
, then it will returnTrue
. If the task is not completed, it will log some information about the task and returnFalse
Parameters: - client (
Elasticsearch
) – A client connection object - task_id (str) – The task id
Return type: - client (
-
curator.helpers.waiters.
wait_for_it
(client, action, task_id=None, snapshot=None, repository=None, index=None, index_list=None, wait_interval=9, max_wait=-1)¶ This function becomes one place to do all
wait_for_completion
type behaviorsParameters: - client (
Elasticsearch
) – A client connection object - action (str) – The action name that will identify how to wait
- task_id (str) – If the action provided a task_id, this is where it must be declared.
- snapshot (str) – The name of the snapshot.
- repository (str) – The Elasticsearch snapshot repository to use
- wait_interval (int) – Seconds to wait between completion checks.
- max_wait (int) – Maximum number of seconds to
wait_for_completion
Return type: - client (