pgtoolkit/toolbase.py
author Radek Brich <radek.brich@devl.cz>
Wed, 28 Mar 2012 17:25:18 +0200
changeset 33 bd0beda49bcb
parent 31 c2e6e24b83d9
child 34 98c7809af415
permissions -rw-r--r--
PgManager: log connection name with queries. BatchUpdateTool: handle some possible exceptions and try reconnect to database.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     1
import argparse
13
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
     2
import logging
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     3
11
bc69eca59041 PgConsole: add app.py, splitted from pgconsole.py.
Radek Brich <radek.brich@devl.cz>
parents: 9
diff changeset
     4
from pgtoolkit import pgmanager, config
13
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
     5
from pgtoolkit.coloredformatter import ColoredFormatter
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
     6
from pgtoolkit.highlight import highlight
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     7
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     8
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
     9
class ConnectionInfoNotFound(Exception):    
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    10
    pass
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    11
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    12
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    13
class ToolBase:
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    14
    def __init__(self, name, desc):
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    15
        self.parser = argparse.ArgumentParser(description=desc)
13
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    16
        self.parser.add_argument('-d', dest='debug', action='store_true',
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    17
            help='Debug mode - print database queries.')
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    18
        
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    19
        self.config = config.ConfigParser()
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    20
        self.config.add_argument('databases', type=dict)
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    21
        self.config.add_argument('meta_db')
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    22
        self.config.add_argument('meta_query')
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    23
        
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    24
        self.pgm = pgmanager.get_instance()
30
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    25
        self.target_isolation_level = None
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    26
        
31
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
    27
    def init(self):
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    28
        self.config.load('pgtoolkit.conf')
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    29
        self.args = self.parser.parse_args()
14
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    30
        self.init_logging()
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    31
    
14
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    32
    def init_logging(self):
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    33
        # logging
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    34
        handler = logging.StreamHandler()
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    35
        format = ColoredFormatter(highlight(1,7,0)+'%(asctime)s %(levelname)-5s'+highlight(0)+' %(message)s', '%H:%M:%S')
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    36
        handler.setFormatter(format)
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    37
        handler.setLevel(logging.DEBUG)
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    38
        self.log = logging.getLogger('main')
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    39
        self.log.addHandler(handler)
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    40
        self.log.setLevel(logging.DEBUG)
a900bc629ecc TableDiffTool: add arguments to set different destination schema and table. PgDataDiff: fixes.
Radek Brich <radek.brich@devl.cz>
parents: 13
diff changeset
    41
        
13
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    42
        if self.args.debug:
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    43
            handler = logging.StreamHandler()
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    44
            handler.setFormatter(format)
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    45
            handler.setLevel(logging.DEBUG)
29
8636719a30f6 Add batchquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 20
diff changeset
    46
            logger = logging.getLogger('pgmanager_sql')
13
16dc5dec9c36 ToolBase: add -d parameter, which triggers debug mode (print SQL queries).
Radek Brich <radek.brich@devl.cz>
parents: 11
diff changeset
    47
            logger.addHandler(handler)
29
8636719a30f6 Add batchquery tool.
Radek Brich <radek.brich@devl.cz>
parents: 20
diff changeset
    48
            logger.setLevel(logging.DEBUG)
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    49
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    50
    def prepare_conn_from_metadb(self, name, lookup_name):
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    51
        '''Create connection in pgmanager using meta DB.
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    52
        
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    53
        name -- Name for connection in pgmanager.
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    54
        lookup_name -- Name of connection in meta DB.
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    55
        
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    56
        '''
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    57
        with self.pgm.cursor('meta') as curs:
17
f768a3529ee7 ToolBase - implement 'database' config option.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    58
            curs.execute(self.config.meta_query, [lookup_name])
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    59
            row = curs.fetchone_dict()
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    60
            curs.connection.commit()
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    61
            if row:
30
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    62
                self.pgm.create_conn(name=name,
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    63
                    isolation_level=self.target_isolation_level,
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    64
                    **row)
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    65
                return True
2
efee419b7a2d Fix pgmanager exceptions.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    66
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    67
    def prepare_conn_from_config(self, name, lookup_name):
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    68
        '''Create connection in pgmanager using info in config.databases.'''
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    69
        if self.config.databases:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    70
            if lookup_name in self.config.databases:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    71
                dsn = self.config.databases[lookup_name]
30
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    72
                self.pgm.create_conn(name=name,
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    73
                    isolation_level=self.target_isolation_level,
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    74
                    dsn=dsn)
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    75
                return True
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    76
31
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
    77
    def prepare_conns_from_cmdline_args(self, *pgm_names):
5
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    78
        if self.config.meta_db:
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    79
            self.pgm.create_conn(name='meta', dsn=self.config.meta_db)
30
a8b7cd92f39f Fix analyzeall tool. Add user output formating to batchquery tool. Add isolation_level setting support to ToolBase.
Radek Brich <radek.brich@devl.cz>
parents: 29
diff changeset
    80
        
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    81
        for name in pgm_names:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    82
            lookup_name = self.args.__dict__[name]
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    83
            found = self.prepare_conn_from_config(name, lookup_name)
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    84
            if not found and self.config.meta_db:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    85
                found = self.prepare_conn_from_metadb(name, lookup_name)
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    86
            if not found:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    87
                raise ConnectionInfoNotFound('Connection name "%s" not found in config nor in meta DB.' % lookup_name)
17
f768a3529ee7 ToolBase - implement 'database' config option.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    88
        
20
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    89
        if self.config.meta_db:
73f0d53fef6b PgManager: Do not add NullHandler to logger. Rewrite get_instance(). ToolBase: fix prepare_conns() method.
Radek Brich <radek.brich@devl.cz>
parents: 17
diff changeset
    90
            self.pgm.close_conn('meta')
5
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    91
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    92
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    93
class SimpleTool(ToolBase):
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    94
    def __init__(self, name, desc):
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    95
        ToolBase.__init__(self, name, desc)        
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    96
        self.parser.add_argument('target', metavar='target', type=str, help='Target database')
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    97
      
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
    98
    def init(self):
31
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
    99
        ToolBase.init(self)
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
   100
        self.prepare_conns_from_cmdline_args('target')
5
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   101
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   102
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   103
class SrcDstTool(ToolBase):
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   104
    def __init__(self, name, desc):
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   105
        ToolBase.__init__(self, name, desc)        
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   106
        self.parser.add_argument('src', metavar='source', type=str, help='Source database')
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   107
        self.parser.add_argument('dst', metavar='destination', type=str, help='Destination database')
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   108
      
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   109
    def init(self):
31
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
   110
        ToolBase.init(self)
c2e6e24b83d9 Add browser - database schema browser using tuikit (curses UI). Add listdepends - tool which shows depending views for column. Update pgdatadiff - allow composite primary key. Update pgmanager - RowDict is now OrderedDict. Drop support for Python2.x.
Radek Brich <radek.brich@devl.cz>
parents: 30
diff changeset
   111
        self.prepare_conns_from_cmdline_args('src', 'dst')
5
57cf8fdff5ed Clean up tools.
Radek Brich <radek.brich@devl.cz>
parents: 2
diff changeset
   112