author | Radek Brich <brich.radek@ifortuna.cz> |
Wed, 07 May 2014 18:33:50 +0200 | |
changeset 102 | fda45bdfd68d |
parent 101 | 2a2d0d5df03b |
permissions | -rw-r--r-- |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
1 |
from pgtoolkit import toolbase, pgmanager, pgdatadiff |
100
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
2 |
from pgtoolkit.toolbase import SrcDstTablesTool |
83
515fadd3d286
Add dependency on pycolib. Move common modules to pycolib. Add example table schema for meta DB.
Radek Brich <radek.brich@devl.cz>
parents:
56
diff
changeset
|
3 |
from pycolib.ansicolor import highlight, BOLD, YELLOW |
6 | 4 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
5 |
import sys |
6 | 6 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
7 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
8 |
class TableDiffTool(SrcDstTablesTool): |
100
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
9 |
|
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
10 |
""" |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
11 |
Print differences between data in tables. |
100
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
12 |
|
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
13 |
Requirements: |
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
14 |
* Source table must have defined PRIMARY KEY. |
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
15 |
* Destination table must contain all columns from source table. |
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
16 |
Order is not important. |
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
17 |
|
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
18 |
""" |
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
19 |
|
6 | 20 |
def __init__(self): |
100
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
21 |
SrcDstTablesTool.__init__(self, name='tablediff', desc=self.__doc__, allow_reverse=True) |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
22 |
|
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
23 |
def specify_args(self): |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
24 |
SrcDstTablesTool.specify_args(self) |
7
685b20d2d3ab
Reorganize directories. PgDataDiff - reworked. PgManager - add fetchone_adapted, fetchall_adapted to cursor.
Radek Brich <radek.brich@devl.cz>
parents:
6
diff
changeset
|
25 |
self.parser.add_argument('--sql', action='store_true', help='Output is SQL script.') |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
26 |
self.parser.add_argument('--rowcount', action='store_true', help='Compare number of rows.') |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
27 |
self.parser.add_argument('-o', '--output-file', help='Output file for sql queries.') |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
28 |
|
6 | 29 |
def main(self): |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
30 |
srcconn = self.pgm.get_conn('src') |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
31 |
dstconn = self.pgm.get_conn('dst') |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
32 |
|
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
33 |
if self.args.output_file: |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
34 |
output_file = open(self.args.output_file, 'w') |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
35 |
else: |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
36 |
output_file = sys.stdout |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
37 |
|
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
38 |
dd = pgdatadiff.PgDataDiff(srcconn, dstconn) |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
39 |
|
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
40 |
for srcschema, srctable, dstschema, dsttable in self.tables(): |
35
e7f79c4a27ce
TableCopyTool: order tables with respect to references.
Radek Brich <radek.brich@devl.cz>
parents:
34
diff
changeset
|
41 |
print('-- Diff from [%s] %s.%s to [%s] %s.%s' % ( |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
42 |
self.args.src, srcschema, srctable, |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
43 |
self.args.dst, dstschema, dsttable), |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
44 |
file=output_file) |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
45 |
|
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
46 |
if self.args.rowcount: |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
47 |
with self.pgm.cursor('src') as curs: |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
48 |
curs.execute('''SELECT count(*) FROM "%s"."%s"''' % (srcschema, srctable)) |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
49 |
srccount = curs.fetchone()[0] |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
50 |
with self.pgm.cursor('dst') as curs: |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
51 |
curs.execute('''SELECT count(*) FROM "%s"."%s"''' % (dstschema, dsttable)) |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
52 |
dstcount = curs.fetchone()[0] |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
53 |
if srccount != dstcount: |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
54 |
print(highlight(1, BOLD | YELLOW), |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
55 |
"Row count differs: src=%s dst=%s" % (srccount, dstcount), |
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
56 |
highlight(0), sep='', file=output_file) |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
57 |
continue |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
58 |
|
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
59 |
dd.settable1(srctable, srcschema) |
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
60 |
dd.settable2(dsttable, dstschema) |
56
94e091c23ebb
Add reverse parameter for diff tools.
Radek Brich <radek.brich@devl.cz>
parents:
41
diff
changeset
|
61 |
|
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
62 |
if self.args.sql: |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
63 |
dd.print_patch(file=output_file) |
34
98c7809af415
Add PgDataCopy. Add TableCopyTool.Add SrcDstTablesTool class to toolbase, use in tablecopy, tablediff.
Radek Brich <radek.brich@devl.cz>
parents:
14
diff
changeset
|
64 |
else: |
101
2a2d0d5df03b
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Radek Brich <brich.radek@ifortuna.cz>
parents:
100
diff
changeset
|
65 |
dd.print_diff(file=output_file) |
6 | 66 |
|
67 |
||
100
d6088dba8fea
Add pgtool wrapper for all tools. Only this script will be installed into system bin.
Radek Brich <brich.radek@ifortuna.cz>
parents:
93
diff
changeset
|
68 |
cls = TableDiffTool |
7
685b20d2d3ab
Reorganize directories. PgDataDiff - reworked. PgManager - add fetchone_adapted, fetchall_adapted to cursor.
Radek Brich <radek.brich@devl.cz>
parents:
6
diff
changeset
|
69 |