pgtoolkit/tools/tablediff.py
author Radek Brich <radek.brich@devl.cz>
Mon, 26 May 2014 18:18:21 +0200
changeset 103 24e94a3da209
parent 101 2a2d0d5df03b
permissions -rw-r--r--
Update bigtables tool: Sort by size with indexes, not just data.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
     1
from pgtoolkit import toolbase, pgmanager, pgdatadiff
100
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
     2
from pgtoolkit.toolbase import SrcDstTablesTool
83
515fadd3d286 Add dependency on pycolib. Move common modules to pycolib. Add example table schema for meta DB.
Radek Brich <radek.brich@devl.cz>
parents: 56
diff changeset
     3
from pycolib.ansicolor import highlight, BOLD, YELLOW
6
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     4
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
     5
import sys
6
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     6
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
     7
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
     8
class TableDiffTool(SrcDstTablesTool):
100
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
     9
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    10
    """
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    11
    Print differences between data in tables.
100
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    12
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    13
    Requirements:
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    14
    * Source table must have defined PRIMARY KEY.
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    15
    * Destination table must contain all columns from source table.
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    16
      Order is not important.
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    17
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    18
    """
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    19
6
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    20
    def __init__(self):
100
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    21
        SrcDstTablesTool.__init__(self, name='tablediff', desc=self.__doc__, allow_reverse=True)
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    22
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    23
    def specify_args(self):
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    24
        SrcDstTablesTool.specify_args(self)
7
685b20d2d3ab Reorganize directories. PgDataDiff - reworked. PgManager - add fetchone_adapted, fetchall_adapted to cursor.
Radek Brich <radek.brich@devl.cz>
parents: 6
diff changeset
    25
        self.parser.add_argument('--sql', action='store_true', help='Output is SQL script.')
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    26
        self.parser.add_argument('--rowcount', action='store_true', help='Compare number of rows.')
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    27
        self.parser.add_argument('-o', '--output-file', help='Output file for sql queries.')
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    28
6
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    29
    def main(self):
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    30
        srcconn = self.pgm.get_conn('src')
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    31
        dstconn = self.pgm.get_conn('dst')
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    32
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    33
        if self.args.output_file:
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    34
            output_file = open(self.args.output_file, 'w')
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    35
        else:
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    36
            output_file = sys.stdout
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    37
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    38
        dd = pgdatadiff.PgDataDiff(srcconn, dstconn)
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    39
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    40
        for srcschema, srctable, dstschema, dsttable in self.tables():
35
e7f79c4a27ce TableCopyTool: order tables with respect to references.
Radek Brich <radek.brich@devl.cz>
parents: 34
diff changeset
    41
            print('-- Diff from [%s] %s.%s to [%s] %s.%s' % (
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    42
                self.args.src, srcschema, srctable,
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    43
                self.args.dst, dstschema, dsttable),
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    44
                file=output_file)
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    45
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    46
            if self.args.rowcount:
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    47
                with self.pgm.cursor('src') as curs:
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    48
                    curs.execute('''SELECT count(*) FROM "%s"."%s"''' % (srcschema, srctable))
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    49
                    srccount = curs.fetchone()[0]
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    50
                with self.pgm.cursor('dst') as curs:
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    51
                    curs.execute('''SELECT count(*) FROM "%s"."%s"''' % (dstschema, dsttable))
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    52
                    dstcount = curs.fetchone()[0]
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    53
                if srccount != dstcount:
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    54
                    print(highlight(1, BOLD | YELLOW),
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    55
                        "Row count differs: src=%s dst=%s" % (srccount, dstcount),
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    56
                        highlight(0), sep='', file=output_file)
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    57
                continue
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    58
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    59
            dd.settable1(srctable, srcschema)
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    60
            dd.settable2(dsttable, dstschema)
56
94e091c23ebb Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents: 41
diff changeset
    61
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    62
            if self.args.sql:
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    63
                dd.print_patch(file=output_file)
34
98c7809af415 Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents: 14
diff changeset
    64
            else:
101
2a2d0d5df03b Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents: 100
diff changeset
    65
                dd.print_diff(file=output_file)
6
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    66
4ab077c93b2d Add table diff tool.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    67
100
d6088dba8fea Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents: 93
diff changeset
    68
cls = TableDiffTool
7
685b20d2d3ab Reorganize directories. PgDataDiff - reworked. PgManager - add fetchone_adapted, fetchall_adapted to cursor.
Radek Brich <radek.brich@devl.cz>
parents: 6
diff changeset
    69