tests/multiprocess.py
author Radek Brich <brich.radek@ifortuna.cz>
Tue, 06 May 2014 18:37:43 +0200
changeset 101 2a2d0d5df03b
parent 78 64c62ac8f65d
child 104 d8ff52a0390f
permissions -rwxr-xr-x
Refactor ToolBase to allow tool composition. Add TableSync tool (composited). Move more tools under pgtool.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     1
#!/usr/bin/env python3
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     2
'''Multiprocessing test.
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     3
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     4
PgManager is partially safe against process forking.
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     5
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     6
You can create connection credentials (create_conn) in main process, then fork
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     7
and continue working with pgmanager in children processes. You can call create_conn again
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     8
in children, but new connection credentials will be accessible only from process
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
     9
where it was created -- no sharing between processes.
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    10
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    11
get_conn and cursor will still work after fork. Connections from parent process will
78
64c62ac8f65d Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents: 25
diff changeset
    12
be forgotten, that cannot be used un child process.
24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    13
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    14
Connections cannot be shared between processes, DO NOT:
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    15
 * get_conn, then fork, then put_conn
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    16
 * get_conn, then fork, then use the connection, then .close()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    17
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    18
Good usage:
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    19
 * create_conn, fork, get_conn, cursor, etc.
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    20
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    21
Basically, you can transfer only information from create_conn(), nothing else.
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    22
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    23
'''
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    25
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    26
import multiprocessing
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    27
78
64c62ac8f65d Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents: 25
diff changeset
    28
from tests.config import Config
24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    29
from pgtoolkit import pgmanager
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    30
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    31
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    32
def sub1(id):
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    33
    with pgm.cursor() as curs:
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    34
        print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid()))
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    35
        print('[%d] update' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    36
        curs.execute('''UPDATE test SET name = 'multi-1' WHERE id=%s''', [id])
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    37
        print('[%d] commit' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    38
        curs.connection.commit()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    39
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    40
def sub2(id):
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    41
    pgm = pgmanager.get_instance()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    42
    with pgm.cursor() as curs:
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    43
        print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid()))
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    44
        print('[%d] update' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    45
        curs.execute('''UPDATE test SET name = 'multi-2' WHERE id=%s''', [id])
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    46
        print('[%d] commit' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    47
        curs.connection.commit()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    48
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    49
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    50
cfg = Config('tests.conf')
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    51
pgm = pgmanager.get_instance()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    52
pgm.create_conn(**cfg)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    53
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    54
with pgm.cursor() as curs:
78
64c62ac8f65d Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents: 25
diff changeset
    55
    print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid()))
24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    56
    print('[%d] insert' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    57
    curs.execute('''INSERT INTO test (name) VALUES ('multi') RETURNING id''')
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    58
    id = curs.fetchone()[0]
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    59
    curs.connection.commit()
78
64c62ac8f65d Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents: 25
diff changeset
    60
24
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    61
    print('[%d] update' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    62
    curs.execute('''UPDATE test SET name = 'multi-main' WHERE id=%s''', [id])
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    63
    # not committed
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    64
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    65
p1 = multiprocessing.Process(target=sub1, args=[id])
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    66
p1.start()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    67
p2 = multiprocessing.Process(target=sub2, args=[id])
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    68
p2.start()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    69
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    70
with pgm.cursor() as curs:
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    71
    print('[%d] commit' % multiprocessing.current_process().pid)
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    72
    curs.connection.commit()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    73
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    74
print('join')
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    75
p1.join()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    76
p2.join()
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    77
5664afa530e5 PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff changeset
    78
print('done')