pgtoolkit/tools/loopquery.py
author Radek Brich <radek.brich@devl.cz>
Mon, 26 May 2014 18:18:21 +0200
changeset 103 24e94a3da209
parent 102 fda45bdfd68d
permissions -rw-r--r--
Update bigtables tool: Sort by size with indexes, not just data.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
     1
from pgtoolkit.toolbase import SimpleTool
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     2
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
     3
import logging.handlers
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     4
import time
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
     5
from datetime import datetime, timedelta
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     6
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     7
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
     8
class LoopQueryTool(SimpleTool):
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
     9
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    10
    """
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    11
    Execute queries in loop, with configurable interval.
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    12
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    13
    """
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    14
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    15
    def __init__(self):
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
    16
        SimpleTool.__init__(self, name='loopquery')
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    17
        self.target_isolation_level = 'autocommit'
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    18
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    19
    def specify_args(self):
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
    20
        SimpleTool.specify_args(self)
65
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    21
        self.parser.add_argument('-q', dest='queries', metavar='QUERY', nargs='*', help='Queries to run.')
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    22
        self.parser.add_argument('--mins', dest='delay_mins', type=int, help='Delay between queries in minutes.')
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    23
        self.parser.add_argument('--secs', dest='delay_secs', type=int, help='Delay between queries in seconds.')
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    24
91
fba99a54ec49 Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 78
diff changeset
    25
        self.config.add_option('queries', type=list, default=[])
fba99a54ec49 Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 78
diff changeset
    26
        self.config.add_option('delay_mins', type=int, default=0)
fba99a54ec49 Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 78
diff changeset
    27
        self.config.add_option('delay_secs', type=int, default=0)
fba99a54ec49 Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 78
diff changeset
    28
        self.config.add_option('log_path', type=str)
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    29
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    30
    def load_args(self, args=None, config_file=None):
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
    31
        SimpleTool.load_args(self, args, config_file)
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    32
        self.queries = self.args.queries or self.config.queries
65
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    33
        self.delay_mins = self.args.delay_mins or self.config.delay_mins
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    34
        self.delay_secs = self.args.delay_secs or self.config.delay_secs
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    35
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    36
    def init_logging(self):
102
fda45bdfd68d Update ToolBase: Load cascade of config files, add -c parameter.
Radek Brich <brich.radek@ifortuna.cz>
parents: 101
diff changeset
    37
        SimpleTool.init_logging(self)
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    38
        if self.config.log_path:
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    39
            self.init_file_logs(self.config.log_path)
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    40
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    41
    def init_file_logs(self, path):
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    42
        format = logging.Formatter('%(asctime)s %(levelname)-5s %(message)s', '%y-%m-%d %H:%M:%S')
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    43
        handler = logging.handlers.TimedRotatingFileHandler(path+'/main.log', when='midnight', backupCount=5)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    44
        handler.setFormatter(format)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    45
        handler.setLevel(logging.DEBUG)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    46
        logging.getLogger('main').addHandler(handler)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    47
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    48
        format = logging.Formatter('%(asctime)s %(message)s', '%y-%m-%d %H:%M:%S')
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    49
        handler = logging.handlers.TimedRotatingFileHandler(path+'/pgnotices.log', when='midnight', backupCount=5)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    50
        handler.setFormatter(format)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    51
        handler.setLevel(logging.DEBUG)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    52
        logging.getLogger('pgmanager_notices').addHandler(handler)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    53
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    54
    def main(self):
65
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    55
        self.reset()
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    56
        while True:
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    57
            self.wait()
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    58
            self.action()
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    59
65
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    60
    def reset(self):
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    61
        """Check current time, set next action time."""
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    62
        dt = datetime.today()
66
3cd28308d67b Fix loopquery timing.
Radek Brich <radek.brich@devl.cz>
parents: 65
diff changeset
    63
        dt = dt.replace(microsecond = 0)
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    64
        self.next_action_time = dt + timedelta(minutes = self.delay_mins,
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    65
                                               seconds = self.delay_secs)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    66
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    67
    def wait(self):
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    68
        """Wait for action time, compute next action time."""
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    69
        now = datetime.today()
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    70
        self.log.debug('Next run %s', self.next_action_time)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    71
        if self.next_action_time > now:
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    72
            td = self.next_action_time - now
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    73
            self.log.debug('Sleep %ds', td.seconds + td.microseconds/1e6)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    74
            time.sleep(td.seconds + td.microseconds/1e6)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    75
        self.next_action_time += timedelta(minutes = self.delay_mins,
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    76
                                           seconds = self.delay_secs)
65
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    77
        # in case that action took too long and next planned time would
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    78
        # be in past -> reset planner
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    79
        if self.next_action_time < now:
ea79c89e9517 Update loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 62
diff changeset
    80
            self.reset()
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    81
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    82
    def action(self):
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    83
        """Execute the queries."""
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    84
        for q in self.queries:
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    85
            self.log.info('%s', q)
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    86
            with self.pgm.cursor('target') as curs:
62
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    87
                curs.execute(q)
af637235ca81 Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents: 51
diff changeset
    88
        self.log.info('Done')
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    89
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    90
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    91
cls = LoopQueryTool
51
bdc44f96cb0b Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    92