author | Radek Brich <radek.brich@devl.cz> |
Fri, 25 Jul 2014 15:15:16 +0200 | |
changeset 106 | db4c582a2abd |
parent 104 | d8ff52a0390f |
permissions | -rwxr-xr-x |
24
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
1 |
#!/usr/bin/env python3 |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
2 |
'''Multiprocessing test. |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
3 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
4 |
PgManager is partially safe against process forking. |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
5 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
6 |
You can create connection credentials (create_conn) in main process, then fork |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
7 |
and continue working with pgmanager in children processes. You can call create_conn again |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
8 |
in children, but new connection credentials will be accessible only from process |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
9 |
where it was created -- no sharing between processes. |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
10 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
11 |
get_conn and cursor will still work after fork. Connections from parent process will |
78
64c62ac8f65d
Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents:
25
diff
changeset
|
12 |
be forgotten, that cannot be used un child process. |
24
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
13 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
14 |
Connections cannot be shared between processes, DO NOT: |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
15 |
* get_conn, then fork, then put_conn |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
16 |
* get_conn, then fork, then use the connection, then .close() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
17 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
18 |
Good usage: |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
19 |
* create_conn, fork, get_conn, cursor, etc. |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
20 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
21 |
Basically, you can transfer only information from create_conn(), nothing else. |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
22 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
23 |
''' |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
24 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
25 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
26 |
import multiprocessing |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
27 |
|
78
64c62ac8f65d
Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents:
25
diff
changeset
|
28 |
from tests.config import Config |
104 | 29 |
from pydbkit import pgmanager |
24
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
30 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
31 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
32 |
def sub1(id): |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
33 |
with pgm.cursor() as curs: |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
34 |
print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid())) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
35 |
print('[%d] update' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
36 |
curs.execute('''UPDATE test SET name = 'multi-1' WHERE id=%s''', [id]) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
37 |
print('[%d] commit' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
38 |
curs.connection.commit() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
39 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
40 |
def sub2(id): |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
41 |
pgm = pgmanager.get_instance() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
42 |
with pgm.cursor() as curs: |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
43 |
print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid())) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
44 |
print('[%d] update' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
45 |
curs.execute('''UPDATE test SET name = 'multi-2' WHERE id=%s''', [id]) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
46 |
print('[%d] commit' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
47 |
curs.connection.commit() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
48 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
49 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
50 |
cfg = Config('tests.conf') |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
51 |
pgm = pgmanager.get_instance() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
52 |
pgm.create_conn(**cfg) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
53 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
54 |
with pgm.cursor() as curs: |
78
64c62ac8f65d
Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents:
25
diff
changeset
|
55 |
print('[%d] pgconn: %d' % (multiprocessing.current_process().pid, curs.connection.get_backend_pid())) |
24
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
56 |
print('[%d] insert' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
57 |
curs.execute('''INSERT INTO test (name) VALUES ('multi') RETURNING id''') |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
58 |
id = curs.fetchone()[0] |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
59 |
curs.connection.commit() |
78
64c62ac8f65d
Rename config module to configparser.
Radek Brich <radek.brich@devl.cz>
parents:
25
diff
changeset
|
60 |
|
24
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
61 |
print('[%d] update' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
62 |
curs.execute('''UPDATE test SET name = 'multi-main' WHERE id=%s''', [id]) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
63 |
# not committed |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
64 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
65 |
p1 = multiprocessing.Process(target=sub1, args=[id]) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
66 |
p1.start() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
67 |
p2 = multiprocessing.Process(target=sub2, args=[id]) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
68 |
p2.start() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
69 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
70 |
with pgm.cursor() as curs: |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
71 |
print('[%d] commit' % multiprocessing.current_process().pid) |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
72 |
curs.connection.commit() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
73 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
74 |
print('join') |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
75 |
p1.join() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
76 |
p2.join() |
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
77 |
|
5664afa530e5
PgManager: Add partial support for multiprocessing.
Radek Brich <radek.brich@devl.cz>
parents:
diff
changeset
|
78 |
print('done') |