Helpers¶
Date Ops¶
- class curator.helpers.date_ops.TimestringSearch(timestring)¶
Bases:
object
An object to allow repetitive search against a string,
searchme
, without having to repeatedly recreate the regex.- Parameters:
timestring (
strftime()
) – Anstrftime
pattern
- get_epoch(searchme)¶
- Parameters:
searchme – A string to be matched against
pattern
that matchestimestring
- Returns:
The epoch timestamp extracted from
searchme
by regex matching againstpattern
- Return type:
int or None
- pattern¶
Object attribute.
re.compile(regex)
whereregex = r'(?P<date>{0})'.format(get_date_regex(timestring))
. Usesget_date_regex()
- timestring¶
Object attribute preserving param
timestring
- curator.helpers.date_ops.absolute_date_range(unit, date_from, date_to, date_from_format=None, date_to_format=None)¶
This function calculates a date range with an absolute time stamp for both the start time and the end time. These dates are converted to epoch time. The parameter
unit
is used when the same simplified date is used for bothdate_from
anddate_to
to calculate the duration. For example, ifunit
ismonths
, anddate_from
anddate_to
are both2017.01
, then the entire month of January 2017 will be the absolute date range.- Parameters:
unit (str) – One of
hours
,days
,weeks
,months
, oryears
.date_from (str) – The simplified date for the start of the range
date_to (str) – The simplified date for the end of the range.
date_from_format (str) – The
strftime()
string used to parsedate_from
date_to_format (str) – The
strftime()
string used to parsedate_to
- Returns:
The epoch start time and end time of a date range
- Return type:
- curator.helpers.date_ops.date_range(unit, range_from, range_to, epoch=None, week_starts_on='sunday')¶
This function calculates a date range with a distinct epoch time stamp of both the start time and the end time in counts of
unit
relative to the time at execution, ifepoch
is not set.If
unit
isweeks
, you can also determine when a week begins usingweek_starts_on
, which can be eithersunday
ormonday
.- Parameters:
unit (str) – One of
hours
,days
,weeks
,months
, oryears
.range_from (int) – Count of
unit
in the past/future is the origin?range_to (int) – Count of
unit
in the past/future is the end point?epoch (int) – An epoch timestamp used to establish a point of reference for calculations.
week_starts_on (str) – Either
sunday
ormonday
. Default issunday
- Returns:
The epoch start time and end time of a date range
- Return type:
- curator.helpers.date_ops.datetime_to_epoch(mydate)¶
Converts datetime into epoch seconds
- curator.helpers.date_ops.fix_epoch(epoch)¶
Fix value of
epoch
to be the count since the epoch in seconds only, which should be 10 or fewer digits long.
- curator.helpers.date_ops.get_date_regex(timestring)¶
- Parameters:
timestring (
strftime()
) – Anstrftime
pattern- Returns:
A regex string based on a provided
strftime()
timestring
.- Return type:
- curator.helpers.date_ops.get_datemath(client, datemath, random_element=None)¶
- Parameters:
client (
Elasticsearch
) – A client connection objectdatemath (
datemath
) – An elasticsearch datemath stringrandom_element (str) – This allows external randomization of the name and is only useful for tests so that you can guarantee the output because you provided the random portion.
- Returns:
the parsed index name from
datemath
- Return type:
- curator.helpers.date_ops.get_datetime(index_timestamp, timestring)¶
- Parameters:
index_timestamp (str) – The index timestamp
timestring (
strftime()
) – Anstrftime
pattern
- Returns:
The datetime extracted from the index name, which is the index creation time.
- Return type:
- curator.helpers.date_ops.get_point_of_reference(unit, count, epoch=None)¶
- Parameters:
unit (str) – One of
seconds
,minutes
,hours
,days
,weeks
,months
, oryears
.unit_count (int) – The number of
units
.unit_count
*unit
will be calculated out to the relative number of seconds.epoch (int) – An epoch timestamp used in conjunction with
unit
andunit_count
to establish a point of reference for calculations.
- Returns:
A point-of-reference timestamp in epoch + milliseconds by deriving from a
unit
and acount
, and an optional reference timestamp,epoch
- Return type:
- curator.helpers.date_ops.get_unit_count_from_name(index_name, pattern)¶
- curator.helpers.date_ops.handle_iso_week_number(mydate, timestring, index_timestamp)¶
- Parameters:
mydate (
datetime
) – A Python datetimetimestring (
strftime()
) – Anstrftime
patternindex_timestamp (str) – The index timestamp
- Returns:
The date of the previous week based on ISO week number
- Return type:
- curator.helpers.date_ops.isdatemath(data)¶
- curator.helpers.date_ops.parse_date_pattern(name)¶
Scan and parse
name
forstrftime()
strings, replacing them with the associated value when found, but otherwise returning lowercase values, as uppercase snapshot names are not allowed. It will detect if the first character is a<
, which would indicatename
is going to be using Elasticsearch date math syntax, and skip accordingly.The
strftime()
identifiers that Curator currently recognizes as acceptable include:Y
: A 4 digit yeary
: A 2 digit yearm
: The 2 digit monthW
: The 2 digit week of the yeard
: The 2 digit day of the monthH
: The 2 digit hour of the day, in 24 hour notationM
: The 2 digit minute of the hourS
: The 2 digit number of second of the minutej
: The 3 digit day of the year
- Parameters:
name (str) – A name, which can contain
strftime()
strings- Returns:
The parsed date pattern
- Return type:
- curator.helpers.date_ops.parse_datemath(client, value)¶
Validate that
value
looks like proper datemath. If it passes this test, then try to ship it to Elasticsearch for real. It may yet fail this test, and if it does, it will raise aConfigurationError
exception. If it passes, return the fully parsed string.- Parameters:
client (
Elasticsearch
) – A client connection objectvalue (str) – A string to check for datemath
- Returns:
A datemath indexname, fully rendered by Elasticsearch
- Return type:
Getters¶
- curator.helpers.getters.byte_size(num, suffix='B')¶
- curator.helpers.getters.get_alias_actions(oldidx, newidx, aliases)¶
- Parameters:
- Returns:
A list of actions suitable for
update_aliases()
actions
kwarg.- Return type:
- curator.helpers.getters.get_data_tiers(client)¶
Get all valid data tiers from the node roles of each node in the cluster by polling each node
- Parameters:
client (
Elasticsearch
) – A client connection object- Returns:
The available data tiers in
tier: bool
form.- Return type:
- curator.helpers.getters.get_indices(client, search_pattern='_all')¶
Calls
indices()
- Parameters:
client (
Elasticsearch
) – A client connection object- Returns:
The current list of indices from the cluster
- Return type:
- curator.helpers.getters.get_repository(client, repository='')¶
Calls
get_repository()
- Parameters:
client (
Elasticsearch
) – A client connection objectrepository (str) – The Elasticsearch snapshot repository to use
- Returns:
Configuration information for
repository
.- Return type:
- curator.helpers.getters.get_snapshot(client, repository=None, snapshot='')¶
Calls
get()
- Parameters:
client (
Elasticsearch
) – A client connection objectrepository (str) – The Elasticsearch snapshot repository to use
snapshot (str) – The snapshot name, or a comma-separated list of snapshots
- Returns:
Information about the provided
snapshot
, a snapshot (or a comma-separated list of snapshots). If no snapshot specified, it will collect info for all snapshots. If none exist, an emptydict
will be returned.- Return type:
- curator.helpers.getters.get_snapshot_data(client, repository=None)¶
Get all snapshots from repository and return a list. Calls
get()
- Parameters:
client (
Elasticsearch
) – A client connection objectrepository (str) – The Elasticsearch snapshot repository to use
- Returns:
The list of all snapshots from
repository
- Return type:
- curator.helpers.getters.get_tier_preference(client, target_tier='data_frozen')¶
Do the tier preference thing in reverse order from coldest to hottest Based on the value of
target_tier
, build out the list to use.- Parameters:
client (
Elasticsearch
) – A client connection objecttarget_tier (str) – The target data tier, e.g. data_warm.
- Returns:
A suitable tier preference string in csv format
- Return type:
- curator.helpers.getters.get_write_index(client, alias)¶
Calls
get_alias()
- Parameters:
client (
Elasticsearch
) – A client connection objectalias (str) – An alias name
- Returns:
The the index name associated with the alias that is designated
is_write_index
- Return type:
- curator.helpers.getters.index_size(client, idx, value='total')¶
Calls
stats()
- Parameters:
client (
Elasticsearch
) – A client connection objectidx (str) – An index name
value (str) – One of either
primaries
ortotal
- Returns:
The sum of either
primaries
ortotal
shards for indexidx
- Return type:
integer
- curator.helpers.getters.name_to_node_id(client, name)¶
Calls
info()
- Parameters:
client (
Elasticsearch
) – A client connection objectname (str) – The node
name
- Returns:
The node_id of the node identified by
name
- Return type:
- curator.helpers.getters.node_id_to_name(client, node_id)¶
Calls
info()
- Parameters:
client (
Elasticsearch
) – A client connection objectnode_id (str) – The node
node_id
- Returns:
The name of the node identified by
node_id
- Return type:
- curator.helpers.getters.node_roles(client, node_id)¶
Calls
info()
- Parameters:
client (
Elasticsearch
) – A client connection objectnode_id (str) – The node
node_id
- Returns:
The list of roles assigned to the node identified by
node_id
- Return type:
- curator.helpers.getters.single_data_path(client, node_id)¶
In order for a shrink to work, it should be on a single filesystem, as shards cannot span filesystems. Calls
stats()
- Parameters:
client (
Elasticsearch
) – A client connection objectnode_id (str) – The node
node_id
- Returns:
True
if the node has a single filesystem, elseFalse
- Return type:
Testers¶
- curator.helpers.testers.ilm_policy_check(client, alias)¶
Test if alias is associated with an ILM policy
Calls
get_settings()
- Parameters:
client (
Elasticsearch
) – A client connection objectalias (str) – The alias name
- Return type:
- curator.helpers.testers.repository_exists(client, repository=None)¶
Calls
get_repository()
- Parameters:
client (
Elasticsearch
) – A client connection objectrepository (str) – The Elasticsearch snapshot repository to use
- Returns:
True
ifrepository
exists, elseFalse
- Return type:
- curator.helpers.testers.rollable_alias(client, alias)¶
Calls
get_alias()
- Parameters:
client (
Elasticsearch
) – A client connection objectalias (str) – An Elasticsearch alias
- Returns:
True
orFalse
depending on whetheralias
is an alias that points to an index that can be used by the_rollover
API.- Return type:
- curator.helpers.testers.snapshot_running(client)¶
Calls
get_repository()
Return
True
if a snapshot is in progress, andFalse
if not- Parameters:
client (
Elasticsearch
) – A client connection object- Return type:
- curator.helpers.testers.validate_actions(data)¶
Validate the
actions
configuration dictionary, as imported from actions.yml, for example.
- curator.helpers.testers.validate_filters(action, myfilters)¶
Validate that myfilters are appropriate for the action type, e.g. no index filters applied to a snapshot list.
- curator.helpers.testers.verify_client_object(test)¶
- Parameters:
test (
Elasticsearch
) – The variable or object to test- Returns:
True
iftest
is a properElasticsearch
client object and raise aTypeError
exception if it is not.- Return type:
- curator.helpers.testers.verify_repository(client, repository=None)¶
Do
verify_repository()
call. If it fails, raise aRepositoryException
.- Parameters:
client (
Elasticsearch
) – A client connection objectrepository (str) – A repository name
- Return type:
None
- curator.helpers.testers.verify_snapshot_list(test)¶
- Parameters:
test (
SnapshotList
) – The variable or object to test- Returns:
None
iftest
is a properSnapshotList
object, else raise aTypeError
exception.- Return type:
None
Utils¶
- curator.helpers.utils.chunk_index_list(indices)¶
This utility chunks very large index lists into 3KB chunks. It measures the size as a csv string, then converts back into a list for the return value.
- curator.helpers.utils.report_failure(exception)¶
Raise a
FailedExecution
exception and include the original error message.- Parameters:
exception (:py:exc:Exception) – The upstream exception.
- Return type:
None
- curator.helpers.utils.show_dry_run(ilo, action, **kwargs)¶
Log dry run output with the action which would have been executed.
Waiters¶
- curator.helpers.waiters.health_check(client, **kwargs)¶
This function calls client.cluster.
health()
and, based on the params provided, will returnTrue
orFalse
depending on whether that particular keyword appears in the output, and has the expected value.If multiple keys are provided, all must match for a
True
response.- Parameters:
client (
Elasticsearch
) – A client connection object- Return type:
- curator.helpers.waiters.relocate_check(client, index)¶
This function calls client.cluster.
state()
with a given index to check if all of the shards for that index are in theSTARTED
state. It will returnTrue
if all primary and replica shards are in theSTARTED
state, and it will returnFalse
if any shard is in a different state.- Parameters:
client (
Elasticsearch
) – A client connection objectindex (str) – The index name
- Return type:
- curator.helpers.waiters.restore_check(client, index_list)¶
This function calls client.indices.
recovery()
with the list of indices to check for complete recovery. It will returnTrue
if recovery of those indices is complete, andFalse
otherwise. It is designed to fail fast: if a single shard is encountered that is still recovering (not inDONE
stage), it will immediately returnFalse
, rather than complete iterating over the rest of the response.- Parameters:
client (
Elasticsearch
) – A client connection objectindex_list (list) – The list of indices to verify having been restored.
- Return type:
- curator.helpers.waiters.snapshot_check(client, snapshot=None, repository=None)¶
This function calls client.snapshot.
get()
and tests to see whether the snapshot is complete, and if so, with what status. It will log errors according to the result. If the snapshot is stillIN_PROGRESS
, it will returnFalse
.SUCCESS
will be anINFO
level message,PARTIAL
nets aWARNING
message,FAILED
is anERROR
, message, and all others will be aWARNING
level message.- Parameters:
client (
Elasticsearch
) – A client connection objectsnapshot (str) – The snapshot name
repository (str) – The repository name
- Return type:
- curator.helpers.waiters.task_check(client, task_id=None)¶
This function calls client.tasks.
get()
with the providedtask_id
. If the task data contains'completed': True
, then it will returnTrue
. If the task is not completed, it will log some information about the task and returnFalse
- Parameters:
client (
Elasticsearch
) – A client connection objecttask_id (str) – The task id
- Return type:
- curator.helpers.waiters.wait_for_it(client, action, task_id=None, snapshot=None, repository=None, index=None, index_list=None, wait_interval=9, max_wait=-1)¶
This function becomes one place to do all
wait_for_completion
type behaviors- Parameters:
client (
Elasticsearch
) – A client connection objectaction (str) – The action name that will identify how to wait
task_id (str) – If the action provided a task_id, this is where it must be declared.
snapshot (str) – The name of the snapshot.
repository (str) – The Elasticsearch snapshot repository to use
wait_interval (int) – Seconds to wait between completion checks.
max_wait (int) – Maximum number of seconds to
wait_for_completion
- Return type:
None