This is just a quick post to advise readers that I have released pgio (The SLOB Method for PostgreSQL) under Apache 2.0. The bits are available at the following link: https://github.com/therealkevinc/pgio/releases/tag/1.0. The README is quite informative.
My last testing before the release showed “out of the box” data loading into Amazon Aurora with PostgreSQL compatibility at a rate of 1.69 TB/h. I only modified the pgio.conf file to specify the connection string and to set scale to 128 GB per schema:
After loading data I edited pgio.conf to increase the number of threads per schema to 16 and then easily drove IOPS to the current, advertised IOPS limit of 120,000 for Amazon Aurora with PostgreSQL compatibility.
Testing PostgreSQL physical I/O on any platform could not be any easier, repeatable, nor understandable.
Enjoy pgio!
I have a problem:
./runit.sh
Date: Wed Oct 30 13:57:46 CET 2019
Database connect string: “pgbench”.
Shared buffers: 1GB.
Testing 4 schemas with 2 thread(s) accessing 100M (12800 blocks) of each schema.
Running iostat, vmstat and mpstat on current host–in background.
Launching sessions. 4 schema(s) will be accessed by 2 thread(s) each.
pg_stat_database stats:
datname| blks_hit| blks_read|tup_returned|tup_fetched|tup_updated
BEFORE: pgbench | 1215174 | 108818 | 742598 | 274296 | 102
AFTER: pgbench | 1215174 | 108818 | 742598 | 274296 | 102
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
It didn’t run. Please share pgio.conf and then also do sh -x ./runit.sh > runit.out 2>&1 and let us see the runit.out
I got this error when I had a .psqlrc file – try prefixing the CONNECT_STRING setting with “-X” in the pgio.conf file; that fixed it for me.
Hi Mr.Kevin, I was trying SLOB for postgres and hit the same error. Could you please help here? ‘setup.sh’ ran fine and loaded approx 400gb data but ‘runit.sh’ is failing,
======
postgres@st-caf-postgresdb pgio-master]$ ./runit.sh 1
Date: Tue Mar 15 10:27:24 UTC 2022
Database connect string: “actdb”.
Shared buffers: 128MB.
Testing 2 schemas with 2 thread(s) accessing 2G (262144 blocks) of each schema.
Running iostat, vmstat and mpstat on current host–in background.
Launching sessions. 2 schema(s) will be accessed by 2 thread(s) each.
pg_stat_database stats:
datname| blks_hit| blks_read|tup_returned|tup_fetched|tup_updated
BEFORE: actdb | 164038048 | 69196715 | 283795857 | 3920392 | 123
AFTER: actdb | 164038048 | 69196715 | 283795857 | 3920392 | 123
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
=====
As suggested by you in the earlier comment, ran” sh -x ./runit.sh > runit.out 2>&1″ and from the runin.out file i see below might be the problem,
=========
++ psql actdb -t –quiet -c ‘select blks_hit from pg_stat_database where datname = ‘\”actdb’\”’
+ after_cache_hits=’ 164040682′
+ run_time=0
+ echo ‘0 69196715 69196715 actdb 2 2 164040682 164040682’
+ f_format_output
+ awk ‘{ printf(“DBNAME: %s. %d schemas, %d threads(each). Run time: %d seconds. RIOPS >%ld%ld<\n\n", $4,$5,$6, $1 , ($2 – $3 ) / $1, ($8 – $7) / $1 ) }'
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
+ /bin/kill -9 4718 4719 4720
===========