author | Radek Brich <brich.radek@ifortuna.cz> |
Tue, 06 May 2014 18:37:43 +0200 | |
changeset 101 | 2a2d0d5df03b |
parent 93 | loopquery.py@b72591087495 |
child 102 | fda45bdfd68d |
permissions | -rw-r--r-- |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
1 |
from pgtoolkit.toolbase import ToolBase |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
2 |
|
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
3 |
import logging.handlers |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
4 |
import time |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
5 |
from datetime import datetime, timedelta |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
6 |
|
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
7 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
8 |
class LoopQueryTool(ToolBase): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
9 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
10 |
""" |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
11 |
Execute queries in loop, with configurable interval. |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
12 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
13 |
""" |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
14 |
|
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
15 |
def __init__(self): |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
16 |
ToolBase.__init__(self, name='loopquery') |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
17 |
self.target_isolation_level = 'autocommit' |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
18 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
19 |
def specify_args(self): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
20 |
ToolBase.specify_args(self) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
21 |
self.parser.add_argument('target', nargs='?', metavar='target', type=str, help='Target database') |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
22 |
self.parser.add_argument('-c', dest='config', type=str, help='Additional config file (besides pgtoolkit.conf).') |
65 | 23 |
self.parser.add_argument('-q', dest='queries', metavar='QUERY', nargs='*', help='Queries to run.') |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
24 |
self.parser.add_argument('--mins', dest='delay_mins', type=int, help='Delay between queries in minutes.') |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
25 |
self.parser.add_argument('--secs', dest='delay_secs', type=int, help='Delay between queries in seconds.') |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
26 |
|
91
fba99a54ec49
Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents:
78
diff
changeset
|
27 |
self.config.add_option('target', type=str, default=None) |
fba99a54ec49
Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents:
78
diff
changeset
|
28 |
self.config.add_option('queries', type=list, default=[]) |
fba99a54ec49
Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents:
78
diff
changeset
|
29 |
self.config.add_option('delay_mins', type=int, default=0) |
fba99a54ec49
Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents:
78
diff
changeset
|
30 |
self.config.add_option('delay_secs', type=int, default=0) |
fba99a54ec49
Add to runquery tool: Parameters for query, output file and format for output file. Fix loopquery tool.
Radek Brich <radek.brich@devl.cz>
parents:
78
diff
changeset
|
31 |
self.config.add_option('log_path', type=str) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
32 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
33 |
def load_args(self, args=None, config_file=None): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
34 |
ToolBase.load_args(self, args, config_file) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
35 |
if self.args.config: |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
36 |
self.config.load(self.args.config) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
37 |
self.queries = self.args.queries or self.config.queries |
65 | 38 |
self.delay_mins = self.args.delay_mins or self.config.delay_mins |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
39 |
self.delay_secs = self.args.delay_secs or self.config.delay_secs |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
40 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
41 |
def init_logging(self): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
42 |
ToolBase.init_logging(self) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
43 |
if self.config.log_path: |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
44 |
self.init_file_logs(self.config.log_path) |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
45 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
46 |
def setup(self, args=None): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
47 |
ToolBase.setup(self, args) |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
48 |
self.prepare_conns(target=self.args.target or self.config.target) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
49 |
|
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
50 |
def init_file_logs(self, path): |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
51 |
format = logging.Formatter('%(asctime)s %(levelname)-5s %(message)s', '%y-%m-%d %H:%M:%S') |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
52 |
handler = logging.handlers.TimedRotatingFileHandler(path+'/main.log', when='midnight', backupCount=5) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
53 |
handler.setFormatter(format) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
54 |
handler.setLevel(logging.DEBUG) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
55 |
logging.getLogger('main').addHandler(handler) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
56 |
|
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
57 |
format = logging.Formatter('%(asctime)s %(message)s', '%y-%m-%d %H:%M:%S') |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
58 |
handler = logging.handlers.TimedRotatingFileHandler(path+'/pgnotices.log', when='midnight', backupCount=5) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
59 |
handler.setFormatter(format) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
60 |
handler.setLevel(logging.DEBUG) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
61 |
logging.getLogger('pgmanager_notices').addHandler(handler) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
62 |
|
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
63 |
def main(self): |
65 | 64 |
self.reset() |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
65 |
while True: |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
66 |
self.wait() |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
67 |
self.action() |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
68 |
|
65 | 69 |
def reset(self): |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
70 |
"""Check current time, set next action time.""" |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
71 |
dt = datetime.today() |
66 | 72 |
dt = dt.replace(microsecond = 0) |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
73 |
self.next_action_time = dt + timedelta(minutes = self.delay_mins, |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
74 |
seconds = self.delay_secs) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
75 |
|
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
76 |
def wait(self): |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
77 |
"""Wait for action time, compute next action time.""" |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
78 |
now = datetime.today() |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
79 |
self.log.debug('Next run %s', self.next_action_time) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
80 |
if self.next_action_time > now: |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
81 |
td = self.next_action_time - now |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
82 |
self.log.debug('Sleep %ds', td.seconds + td.microseconds/1e6) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
83 |
time.sleep(td.seconds + td.microseconds/1e6) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
84 |
self.next_action_time += timedelta(minutes = self.delay_mins, |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
85 |
seconds = self.delay_secs) |
65 | 86 |
# in case that action took too long and next planned time would |
87 |
# be in past -> reset planner |
|
88 |
if self.next_action_time < now: |
|
89 |
self.reset() |
|
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
90 |
|
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
91 |
def action(self): |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
92 |
"""Execute the queries.""" |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
93 |
for q in self.queries: |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
94 |
self.log.info('%s', q) |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
95 |
with self.pgm.cursor('target') as curs: |
62
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
96 |
curs.execute(q) |
af637235ca81
Update loopquery: allow any number of queries, support reading parameters from config file.
Radek Brich <radek.brich@devl.cz>
parents:
51
diff
changeset
|
97 |
self.log.info('Done') |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
98 |
|
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
99 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
100 |
cls = LoopQueryTool |
51
bdc44f96cb0b
Add loopquery tool. Log notices to console from all tools (toolbase).
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
101 |