Nabble - PostgreSQL - performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T15:01:41Z

If Autovacuum was working, and your tables still got very bloated, it may be because your free space map is not configured large enough.
What is your value for max_fsm_pages?

The effect of having max_fsm_pages or max_fsm_relations too small is bloating of tables and indexes.

Increasing it too large will not use a lot of memory. Since you had over 130000 free pages in just one table below, make sure it is set to at least 150000. This may be overkill with regular vacuum or autovacuum, however its much better to have this too large than too small.

Your server has 2GB of RAM? You should make sure your shared_buffers is between 100MB and 400MB if this is a dedicated server, or more in some cases.

If you can, plan to migrate to 8.3 (or 8.4 early next year). Vacuum, bloating, and configuration related to these have improved a great deal since 8.1.

Although your Indexes below remain bloated, the fact that they have been vacuumed, combined with a large enough value set in max_fsm_pages, means that they should not get any larger. I am not sure how to foce these to be smaller.
The larger size, after a vacuum and large enough max_fsm_pages value will also not cause a performance problem. It will waste some disk space and be a bit slower to access, but only very slightly.

-----Original Message-----
From: pgsql-performance-owner@... [mailto:pgsql-performance-owner@...] On Behalf Of Andrus
Sent: Friday, November 21, 2008 1:51 PM
To: Richard Huxton
Cc: PFC; pgsql-performance@...
Subject: Re: [PERFORM] Hash join on int takes 8..114 seconds

> If it's not a million rows, then the table is bloated. Try (as postgres
> or some other db superuser) "vacuum full pg_shdepend" and a "reindex
> pg_shdepend".

reindex table pg_shdepend causes error

ERROR: shared table "pg_shdepend" can only be reindexed in stand-alone mode

vacuum full verbose pg_shdepend seems to work but indexes are still bloated.
How to remove index bloat ?

sizes after vacuum full are below.
pg_shdepend size 1234 MB includes its index sizes, so indexes are 100%
bloated.

4 1214 pg_catalog.pg_shdepend 1234 MB
6 1232 pg_catalog.pg_shdepend_depender_index 795 MB
7 1233 pg_catalog.pg_shdepend_reference_index 439 MB

Andrus.

vacuum full verbose pg_shdepend;

INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 254 removable, 3625 nonremovable row versions in
131517 pages
DETAIL: 0 dead row versions cannot be removed yet.
Nonremovable row versions range from 49 to 49 bytes long.
There were 16115259 unused item pointers.
Total free space (including removable row versions) is 1010091872 bytes.
131456 pages are or will become empty, including 8 at the end of the table.
131509 pages containing 1010029072 free bytes are potential move
destinations.
CPU 2.08s/0.92u sec elapsed 63.51 sec.
INFO: index "pg_shdepend_depender_index" now contains 3625 row versions in
101794 pages
DETAIL: 254 index row versions were removed.
101611 index pages have been deleted, 20000 are currently reusable.
CPU 0.87s/0.28u sec elapsed 25.44 sec.
INFO: index "pg_shdepend_reference_index" now contains 3625 row versions in
56139 pages
DETAIL: 254 index row versions were removed.
56076 index pages have been deleted, 20000 are currently reusable.
CPU 0.51s/0.15u sec elapsed 23.10 sec.
INFO: "pg_shdepend": moved 1518 row versions, truncated 131517 to 25 pages
DETAIL: CPU 5.26s/2.39u sec elapsed 89.93 sec.
INFO: index "pg_shdepend_depender_index" now contains 3625 row versions in
101794 pages
DETAIL: 1518 index row versions were removed.
101609 index pages have been deleted, 20000 are currently reusable.
CPU 0.94s/0.28u sec elapsed 24.61 sec.
INFO: index "pg_shdepend_reference_index" now contains 3625 row versions in
56139 pages
DETAIL: 1518 index row versions were removed.
56088 index pages have been deleted, 20000 are currently reusable.
CPU 0.54s/0.14u sec elapsed 21.11 sec.

Query returned successfully with no result in 253356 ms

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T13:51:24Z

> If it's not a million rows, then the table is bloated. Try (as postgres
> or some other db superuser) "vacuum full pg_shdepend" and a "reindex
> pg_shdepend".

reindex table pg_shdepend causes error

ERROR: shared table "pg_shdepend" can only be reindexed in stand-alone mode

vacuum full verbose pg_shdepend seems to work but indexes are still bloated.
How to remove index bloat ?

sizes after vacuum full are below.
pg_shdepend size 1234 MB includes its index sizes, so indexes are 100%
bloated.

4 1214 pg_catalog.pg_shdepend 1234 MB
6 1232 pg_catalog.pg_shdepend_depender_index 795 MB
7 1233 pg_catalog.pg_shdepend_reference_index 439 MB

Andrus.

vacuum full verbose pg_shdepend;

INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 254 removable, 3625 nonremovable row versions in
131517 pages
DETAIL: 0 dead row versions cannot be removed yet.
Nonremovable row versions range from 49 to 49 bytes long.
There were 16115259 unused item pointers.
Total free space (including removable row versions) is 1010091872 bytes.
131456 pages are or will become empty, including 8 at the end of the table.
131509 pages containing 1010029072 free bytes are potential move
destinations.
CPU 2.08s/0.92u sec elapsed 63.51 sec.
INFO: index "pg_shdepend_depender_index" now contains 3625 row versions in
101794 pages
DETAIL: 254 index row versions were removed.
101611 index pages have been deleted, 20000 are currently reusable.
CPU 0.87s/0.28u sec elapsed 25.44 sec.
INFO: index "pg_shdepend_reference_index" now contains 3625 row versions in
56139 pages
DETAIL: 254 index row versions were removed.
56076 index pages have been deleted, 20000 are currently reusable.
CPU 0.51s/0.15u sec elapsed 23.10 sec.
INFO: "pg_shdepend": moved 1518 row versions, truncated 131517 to 25 pages
DETAIL: CPU 5.26s/2.39u sec elapsed 89.93 sec.
INFO: index "pg_shdepend_depender_index" now contains 3625 row versions in
101794 pages
DETAIL: 1518 index row versions were removed.
101609 index pages have been deleted, 20000 are currently reusable.
CPU 0.94s/0.28u sec elapsed 24.61 sec.
INFO: index "pg_shdepend_reference_index" now contains 3625 row versions in
56139 pages
DETAIL: 1518 index row versions were removed.
56088 index pages have been deleted, 20000 are currently reusable.
CPU 0.54s/0.14u sec elapsed 21.11 sec.

Query returned successfully with no result in 253356 ms

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T13:22:55Z

> 2. Run the following commands periodically in this order:
>
> VACUUM FULL;
> vacuum full pg_shdepend;
> CLUSTER rid on (toode);
> CLUSTER dok on (kuupaev);
> REINDEX DATABASE mydb;
> REINDEX SYSTEM mydb;
> ANALYZE;
>
> Are all those command required or can something leaved out ?

Running CLUSTER after VACUUM FULL is just a waste of time. In my
experience CLUSTER is actually faster in case of such heavily bloated
tables - I think this is caused by the fact that it creates indexes from
the beginning instead of updating them (as VACUUM FULL does).

So CLUSTER actually performs REINDEX, so I'd just run

CLUSTER rid ON rid_pkey;
CLUSTER dok ON dok_pkey;
ANALYZE rid;
ANALYZE dok;

Clustering by other indexes might give better performance, using primary
keys is just a safe guess here. This should improve the performance of
your query and it seems these two tables are the most bloated ones.

I wouldn't do the same maintenance on the other tables now - it's just a
waste of time.

>
>> Several other things to consider:
>>
>> 1) Regarding the toode column - why are you using CHAR(20) when the
>> values
>> are actually shorter? This may significantly increase the amount of space
>> required.
>
> There may be some products whose codes may be up to 20 characters.
> PostgreSQL does not hold trailing spaces in db, so this does *not*
> affect to
> space.

OK, I haven't realized this. You're right.

>> 2) I've noticed the CPU used is Celeron, which may negatively affect the
>> speed of hash computation. I'd try to replace it by something faster -
>> say
>> INTEGER as an artificial primary key of the "toode" table and using it as
>> a FK in other tables. This might improve the "Bitmap Heap Scan on rid"
>> part, but yes - it's just a minor improvement compared to the "Hash Join"
>> part of the query.
>
> Natural key Toode CHAR(20) is used widely in different queries.
> Replacing it with INT surrogate key requires major application rewrite.
>
> Should I add surrogate index INT columns to toode and rid table and measure
> test query speed in this case?

Test it. Create tables with fake data, and compare the performance with
and without the surrogate keys. Using a simple data type instead of text
gave me huge performance boost. For example one of my colleagues used
VARCHAR(15) to store IP addresses, and then used them to join tables
(and suffered by the poor perfomance). Replacing them by INET improved
the speed by several orders of magnitude.

>> Materialized views seem like a good idea to me, but maybe I'm not seeing
>> something. What do you mean by "reports are different"? If there is a lot
>> of rows for a given product / day, then creating an aggregated table with
>> (product code / day) as a primary key is quite simple. It may require a
>> lot of disk space, but it'll remove the hash join overhead. But if the
>> queries are very different, then it may be difficult to build such
>> materialized view(s).
>
> log file seems that mostly only those queries are slow:
>
> SELECT ...
> FROM dok JOIN rid USING (dokumnr)
> JOIN ProductId USING (ProductId)
> WHERE rid.ProductId LIKE :p1 || '%' AND dok.SaleDate>=:p2
>
> :p1 and :p2 are parameters different for different queries.
>
> dok contains several years of data. :p2 is usually only few previous months
> or last year ago.
> SELECT column list contains fixed list of known columns from all tables.
>
> How to create index or materialized view to optimize this types of
> queries ?

Well, difficult to answer without detailed information about the queries
you want to run, aggregated values, etc. Materialized views is a world
on it's own, and the best solution depends on (for example):

1) what aggregated values are you interested in (additive values are the
most primitive ones, while VARIANCE etc. make it difficult)

2) do you need current data, or is it OK that today's data are not
available till midnight (for example)?

Another thing you have to consider is whether you want to create
materialized view with final or intermediary data and then compute the
final data somehow (for example monthly totals from daily totals).

The most primitive (but often sufficient) solution is recreating the
materialized view periodically (for example every midnight). In your
case it'd mean running something like

CREATE TABLE materialized_view AS SELECT ... your query here ...
GROUP BY productId, saleDate

This gives you daily totals for each product - the clients then can run
another query to compute the final data.

But of course, if you need to maintain 'current' data, you may create a
set of triggers to update the materialized view. Either after each
modification or (more sophisticated) when it's needed.

See for example a great presentation from this year's PGCon:

http://www.pgcon.org/2008/schedule/events/69.en.html

regards
Tomas

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T12:48:21Z

> Thank you.
> My 8.1.4 postgresql.conf does not contain such option. So
> vacuum_cost_delay is off probably.
> Since doc does not recommend any value, I planned to use 2000
>
> Will value of 30 allow other clients to work when VACUUM FULL is running ?

No, as someone already noted the VACUUM FULL is blocking anyway (and
does not use this value at all).

> Uncommented relevant values in postgresql.conf file are:
>
> shared_buffers = 15000
> work_mem = 512

I'd consider increasing this value a little - 0.5 MB seems too low to me
(but not necessarily).

> maintenance_work_mem = 131072
> fsync = on
> effective_cache_size= 70000

Well, your server has 2GB of RAM and usually it's recommended to set
this value to about 60-70% of your RAM, so using 540MB (25%) seems quite
low. Anyway this is just a hint to PostgreSQL, it does not increase
memory consumption or so - it's just an estimate of how much data are
cached by kernel.

Anyway, I don't expect these values have significant effect in case of
the issue solved in this thread.

regards
Tomas

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T12:20:36Z

Andrus wrote:

>> So I gather you're not doing any vacuuming, eh?
>
> Log files for every day are full of garbage messages below.
> So I hope that vacuum is running well, isn't it ?

This does not really mean that autovacuum has done anything in the
databases. If the times are consistently separated by 1 min, then it's
possible that it always exits without doing anything.

In such old a release we didn't have any decent logging mechanism in
autovacuum :-( You can change log_min_messages to debug2 to see if it
actually does anything or not.

I suggest you connect to the problem database (and then to all others,
just to be sure) and run "vacuum" (no full).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T12:08:45Z

Alvaro,

> 1. vacuum_cost_delay does not affect vacuum full
> 2. vacuum full is always blocking, regardless of settings

So only way is to disable other database acces if vacuum full is required.

> So I gather you're not doing any vacuuming, eh?

Log files for every day are full of garbage messages below.
So I hope that vacuum is running well, isn't it ?

Andrus.

2008-11-19 00:00:48 EET 11728 1 LOG: autovacuum: processing database
"postgres"
2008-11-19 00:01:48 EET 11729 1 LOG: autovacuum: processing database
"mydb1"
2008-11-19 00:02:48 EET 11730 1 LOG: autovacuum: processing database
"emydb1"
2008-11-19 00:03:48 EET 11731 1 LOG: autovacuum: processing database
"template1"
2008-11-19 00:04:48 EET 11732 1 LOG: autovacuum: processing database
"testmydb1"
2008-11-19 00:05:48 EET 11733 1 LOG: autovacuum: processing database
"mydb3"
2008-11-19 00:06:48 EET 11734 1 LOG: autovacuum: processing database
"postgres"
2008-11-19 00:07:48 EET 11735 1 LOG: autovacuum: processing database
"mydb1"
2008-11-19 00:08:48 EET 11736 1 LOG: autovacuum: processing database
"emydb1"
2008-11-19 00:09:48 EET 11737 1 LOG: autovacuum: processing database
"template1"
2008-11-19 00:10:48 EET 11750 1 LOG: autovacuum: processing database
"testmydb1"
2008-11-19 00:11:48 EET 11751 1 LOG: autovacuum: processing database
"mydb3"
2008-11-19 00:12:48 EET 11752 1 LOG: autovacuum: processing database
"postgres"
2008-11-19 00:13:48 EET 11753 1 LOG: autovacuum: processing database
"mydb1"
2008-11-19 00:14:48 EET 11754 1 LOG: autovacuum: processing database
"emydb1"
2008-11-19 00:15:48 EET 11755 1 LOG: autovacuum: processing database
"template1"
2008-11-19 00:16:48 EET 11756 1 LOG: autovacuum: processing database
"testmydb1"
2008-11-19 00:17:48 EET 11757 1 LOG: autovacuum: processing database
"mydb3"
...

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T12:07:02Z

PFC <lists@...> writes:
> Index on orders_products( product_id ) and orders_products( order_id ):
> => Same plan

> Note that in this case, a smarter planner would use the new index to
> perform a BitmapAnd before hitting the heap to get the rows.

Considering that the query has no constraint on
orders_products.order_id, I'm not sure what you think the extra index is
supposed to be used *for*.

(Well, we could put orders as the outside of a nestloop and then we'd
have such a constraint, but with 30000 orders rows to process that plan
would lose big.)

(And yes, the planner did consider such a plan along the way.
See choose_bitmap_and.)

regards, tom lane

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:48:05Z

Andrus wrote:

> Will value of 30 allow other clients to work when VACUUM FULL is running ?

1. vacuum_cost_delay does not affect vacuum full
2. vacuum full is always blocking, regardless of settings

So I gather you're not doing any vacuuming, eh?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:45:11Z

Alvaro,

> Are you really using vacuum_cost_delay=2000? If so, therein lies your
> problem. That's a silly value to use for that variable. Useful values
> are in the 20-40 range probably, or maybe 10-100 being extremely
> generous.

Thank you.
My 8.1.4 postgresql.conf does not contain such option. So vacuum_cost_delay
is off probably.
Since doc does not recommend any value, I planned to use 2000

Will value of 30 allow other clients to work when VACUUM FULL is running ?

Uncommented relevant values in postgresql.conf file are:

shared_buffers = 15000
work_mem = 512
maintenance_work_mem = 131072
fsync = on
effective_cache_size= 70000
log_min_duration_statement= 30000

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:16:46Z

Andrus wrote:

> I discovered vacuum_cost_delay=2000 option. Will this remove blocking
> issue and allow vacuum full to work ?

No.

Are you really using vacuum_cost_delay=2000? If so, therein lies your
problem. That's a silly value to use for that variable. Useful values
are in the 20-40 range probably, or maybe 10-100 being extremely
generous.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:13:46Z

On Friday 21 November 2008, "Andrus" <kobruleht2@...> wrote:
> Those commands cause server probably to stop responding to other client
> like vacuum full pg_shdepend
> did.
>
> Should vacuum_cost_delay = 2000 allow other users to work when running
> those commands ?

Any vacuum full or cluster will lock out other clients. A high
vacuum_cost_delay will just make the vacuum run slower.

--
Corporations will ingest natural resources and defecate garbage until all
resources are depleted, debt can no longer be repaid and our money becomes
worthless - Jay Hanson

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:10:21Z

>> How to vacuum full pg_shdepend automatically so that other users can
>> work at same time ?
>
> Your table is horribly bloated.
> You must use VACUUM FULL + REINDEX (as superuser) on it, however
> unfortunately, it is blocking.
> Therefore, you should wait for sunday night to do this, when noone will
> notice.

Shops are closed late night for a short time, including sunday night.
This time may be shorter than time required to complete VACUUM command.

I discovered vacuum_cost_delay=2000 option. Will this remove blocking issue
and allow vacuum full to work ?

> Meanwhile, you can always VACUUM it (as superuser) and REINDEX it.

I expect that autovacuum does this automatically.

> And while you're at it, VACUUM FULL + reindex the entire database.
> To avoid such annoyances in the future, you should ensure that autovacuum
> runs properly ; you should investigate this. If you use a cron'ed VACUUM
> that does not run as superuser, then it will not be able to VACUUM the
> system catalogs, and the problem will come back.

autovacuum is turned on in postgresql.conf file
log file shows a lot of messages every day that database is vacuumed.
I assume that it is running as user postgres.

I do'nt understand how autovacuum can avoid this: it does not perform vacuum
full so pg_shdepend ja my tables become
bloated again and again.

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:08:27Z

> log file seems that mostly only those queries are slow:
>
> SELECT ...
> FROM dok JOIN rid USING (dokumnr)
> JOIN ProductId USING (ProductId)
> WHERE rid.ProductId LIKE :p1 || '%' AND dok.SaleDate>=:p2
>
> :p1 and :p2 are parameters different for different queries.
>
> dok contains several years of data. :p2 is usually only few previous
> months
> or last year ago.
> SELECT column list contains fixed list of known columns from all tables.
>
> How to create index or materialized view to optimize this types of
> queries ?
>

I would remove some granularity, for instance create a summary table
(materialized view) by month :

- date (contains the first day of the month)
- product_id
- total quantity, total price sold in given month

You get the idea.
If your products belong to categories, and you make queries on all the
products in a category, it could be worth making a summary table for
categories also.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T11:00:09Z

Thomas,

Thank you.

> Just the most important points:
>
> 1) "dok" table contains 1235086 row versions in 171641 pages (with 8kB
> pages this means 1.4GB MB of data), but there are 1834279 unused item
> pointers (i.e. about 60% of the space is wasted)
>
> 2) "rid" table contains 3275189 roiws in 165282 (with 8kB pages this means
> about 1.3GB of data), but there are 1878923 unused item pointers (i.e.
> about 30% of the space is wasted)
>
> 3) don't forget to execute analyze after vacuuming (or vacuum analyze)

autovacuum is running.
So if I understand properly, I must ran
VACUUM FULL ANALYZE dok;
VACUUM FULL ANALYZE rid;

Those commands cause server probably to stop responding to other client like
vacuum full pg_shdepend
did.

Should vacuum_cost_delay = 2000 allow other users to work when running those
commands ?

> 4) I'm not sure why the sizes reported by you (for example 2.3GB vs 1.5GB
> for "doc" table) - the difference seems too large for me.

I used pg_total_relation_size(). So 2.3 GB includes indexes also:

8 44286 dok_tasudok_idx 245 MB
10 44283 dok_klient_idx 142 MB
18 44288 dok_tasumata_idx 91 MB
19 44289 dok_tellimus_idx 89 MB
20 44284dok_krdokumnr_idx 89 MB
21 44285 dok_kuupaev_idx 84 MB
22 43531 makse_pkey 77 MB
23 43479 dok_pkey 74 MB
24 44282 dok_dokumnr_idx 74 MB
26 18663923 dok_yksus_pattern_idx 43 MB
27 18801591 dok_sihtyksus_pattern_idx 42 MB

> Anyway the amount of wasted rows seems significant to me - I'd try to
> solve this first. Either by VACUUM FULL or by CLUSTER. The CLUSTER will
> lock the table exclusively, but the results may be better (when sorting by
> a well chosen index). Don't forget to run ANALYZE afterwards.

How to invoke those commands so that other clients can continue work?
I'm using 8.1.4.
Log files show that autovacuum is running.

I'm planning the following solution:

1. Set

vacuum_cost_delay=2000

2. Run the following commands periodically in this order:

VACUUM FULL;
vacuum full pg_shdepend;
CLUSTER rid on (toode);
CLUSTER dok on (kuupaev);
REINDEX DATABASE mydb;
REINDEX SYSTEM mydb;
ANALYZE;

Are all those command required or can something leaved out ?

> Several other things to consider:
>
> 1) Regarding the toode column - why are you using CHAR(20) when the values
> are actually shorter? This may significantly increase the amount of space
> required.

There may be some products whose codes may be up to 20 characters.
PostgreSQL does not hold trailing spaces in db, so this does *not* affect to
space.

> 2) I've noticed the CPU used is Celeron, which may negatively affect the
> speed of hash computation. I'd try to replace it by something faster - say
> INTEGER as an artificial primary key of the "toode" table and using it as
> a FK in other tables. This might improve the "Bitmap Heap Scan on rid"
> part, but yes - it's just a minor improvement compared to the "Hash Join"
> part of the query.

Natural key Toode CHAR(20) is used widely in different queries. Replacing it
with
INT surrogate key requires major application rewrite.

Should I add surrogate index INT columns to toode and rid table and measure
test query speed in this case?

> Materialized views seem like a good idea to me, but maybe I'm not seeing
> something. What do you mean by "reports are different"? If there is a lot
> of rows for a given product / day, then creating an aggregated table with
> (product code / day) as a primary key is quite simple. It may require a
> lot of disk space, but it'll remove the hash join overhead. But if the
> queries are very different, then it may be difficult to build such
> materialized view(s).

log file seems that mostly only those queries are slow:

SELECT ...
FROM dok JOIN rid USING (dokumnr)
JOIN ProductId USING (ProductId)
WHERE rid.ProductId LIKE :p1 || '%' AND dok.SaleDate>=:p2

:p1 and :p2 are parameters different for different queries.

dok contains several years of data. :p2 is usually only few previous months
or last year ago.
SELECT column list contains fixed list of known columns from all tables.

How to create index or materialized view to optimize this types of queries ?

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T10:31:37Z

> Server has 2 GB RAM.
> It has SATA RAID 0,1 integrated controller (1.5Gbps) and SAMSUNG HD160JJ
> mirrored disks.

You could perhaps run a little check on the performance of the RAID, is
it better than linux software RAID ?
Does it leverage NCQ appropriately when running queries in parallel ?

> -- Receipt headers:
> DOK ( dokumnr INT SERIAL PRIMARY KEY,
> kuupaev DATE --- sales date
> )
> -- Receipt details
> RID ( dokumnr INT,
> toode CHAR(20), -- item code
> CONSTRAINT rid_dokumnr_fkey FOREIGN KEY (dokumnr) REFERENCES dok
> (dokumnr),
> CONSTRAINT rid_toode_fkey FOREIGN KEY (toode)
> REFERENCES firma2.toode (toode)
> )
> -- Products
> TOODE (
> toode CHAR(20) PRIMARY KEY
> )

OK so pretty straightforward :

dok <-(dokumnr)-> rid <-(toode)-> toode

toode.toode should really be an INT though.

>> explain analyze
>> SELECT sum(1)
>> FROM dok JOIN rid USING (dokumnr)
>> JOIN toode USING (toode)
>> LEFT JOIN artliik using(grupp,liik)
>> WHERE rid.toode='X05' AND dok.kuupaev>='2008-09-01'

By the way, note that the presence of the toode table in the query above
is not required at all, unless you use columns of toode in your aggregates.

Let's play with that, after all, it's friday night.

BEGIN;
CREATE TABLE orders (order_id INTEGER NOT NULL, order_date DATE NOT NULL);
CREATE TABLE products (product_id INTEGER NOT NULL, product_name TEXT NOT
NULL);
CREATE TABLE orders_products (order_id INTEGER NOT NULL, product_id
INTEGER NOT NULL, padding1 TEXT, padding2 TEXT);

INSERT INTO products SELECT n, 'product number ' || n::TEXT FROM
generate_series(1,40000) AS n;
INSERT INTO orders SELECT n,'2000-01-01'::date + (n/1000 * '1
DAY'::interval) FROM generate_series(1,1000000) AS n;

SET work_mem TO '1GB';
INSERT INTO orders_products SELECT
a,b,'aibaifbaurgbyioubyfazierugybfoaybofauez',
'hfohbdsqbhjhqsvdfiuazvfgiurvgazrhbazboifhaoifh'
FROM (SELECT DISTINCT (1+(n/10))::INTEGER AS a,
(1+(random()*39999))::INTEGER AS b FROM generate_series( 1,9999999 ) AS n)
AS x;

DELETE FROM orders_products WHERE product_id NOT IN (SELECT product_id
FROM products);
DELETE FROM orders_products WHERE order_id NOT IN (SELECT order_id FROM
orders);
ALTER TABLE orders ADD PRIMARY KEY (order_id);
ALTER TABLE products ADD PRIMARY KEY (product_id);
ALTER TABLE orders_products ADD PRIMARY KEY (order_id,product_id);
ALTER TABLE orders_products ADD FOREIGN KEY (product_id) REFERENCES
products( product_id ) ON DELETE CASCADE;
ALTER TABLE orders_products ADD FOREIGN KEY (order_id) REFERENCES orders(
order_id ) ON DELETE CASCADE;
CREATE INDEX orders_date ON orders( order_date );
COMMIT;
SET work_mem TO DEFAULT;
ANALYZE;

With the following query :

EXPLAIN ANALYZE SELECT sum(1)
FROM orders
JOIN orders_products USING (order_id)
JOIN products USING (product_id)
WHERE orders.order_date BETWEEN '2000-01-01' AND '2000-02-01'
AND products.product_id = 12345;

I get the following results :

orders_products has a PK index on (order_id, product_id). I dropped it.

No index on orders_products :
=> Big seq scan (16 seconds)

Index on orders_products( product_id ) :
Aggregate (cost=2227.22..2227.23 rows=1 width=0) (actual
time=108.204..108.205 rows=1 loops=1)
-> Nested Loop (cost=1312.30..2227.20 rows=7 width=0) (actual
time=105.929..108.191 rows=6 loops=1)
-> Index Scan using products_pkey on products (cost=0.00..8.27
rows=1 width=4) (actual time=0.010..0.014 rows=1 loops=1)
Index Cond: (product_id = 12345)
-> Hash Join (cost=1312.30..2218.85 rows=7 width=4) (actual
time=105.914..108.167 rows=6 loops=1)
Hash Cond: (orders_products.order_id = orders.order_id)
-> Bitmap Heap Scan on orders_products (cost=6.93..910.80
rows=232 width=8) (actual time=0.194..2.175 rows=246 loops=1)
Recheck Cond: (product_id = 12345)
-> Bitmap Index Scan on orders_products_product_id
(cost=0.00..6.87 rows=232 width=0) (actual time=0.129..0.129 rows=246
loops=1)
Index Cond: (product_id = 12345)
-> Hash (cost=949.98..949.98 rows=28432 width=4) (actual
time=105.696..105.696 rows=31999 loops=1)
-> Index Scan using orders_date on orders
(cost=0.00..949.98 rows=28432 width=4) (actual time=0.059..64.443
rows=31999 loops=1)
Index Cond: ((order_date >= '2000-01-01'::date)
AND (order_date <= '2000-02-01'::date))
Total runtime: 108.357 ms
(don't trust this timing, it's a bit cached, this is the same plan as you
get)

Index on orders_products( product_id ) and orders_products( order_id ):
=> Same plan

Note that in this case, a smarter planner would use the new index to
perform a BitmapAnd before hitting the heap to get the rows.

Index on ( order_id, product_id ), orders_products( product_id ):
Index on ( order_id, product_id ):
=> Different plan, slower (especially in second case).

If a "order_date" column is added to the "orders_products" table to make
it into some kind of materialized view :

CREATE TABLE orders_products2 AS SELECT orders.order_id,
orders.order_date, product_id FROM orders JOIN orders_products USING
(order_id);

And an index is created on (product_id, order_date) we get this :

Aggregate (cost=100.44..100.45 rows=1 width=0) (actual time=0.176..0.177
rows=1 loops=1)
-> Nested Loop (cost=0.00..100.42 rows=7 width=0) (actual
time=0.083..0.168 rows=6 loops=1)
-> Index Scan using products_pkey on products (cost=0.00..8.27
rows=1 width=4) (actual time=0.012..0.013 rows=1 loops=1)
Index Cond: (product_id = 12345)
-> Nested Loop (cost=0.00..92.08 rows=7 width=4) (actual
time=0.068..0.147 rows=6 loops=1)
-> Index Scan using orders_products2_pid_date on
orders_products2 (cost=0.00..33.50 rows=7 width=8) (actual
time=0.053..0.076 rows=6 loops=1)
Index Cond: ((product_id = 12345) AND (order_date >=
'2000-01-01'::date) AND (order_date <= '2000-02-01'::date))
-> Index Scan using orders_pkey on orders
(cost=0.00..8.36 rows=1 width=4) (actual time=0.008..0.009 rows=1 loops=6)
Index Cond: (orders.order_id =
orders_products2.order_id)
Total runtime: 0.246 ms

An index on (order_date,product_id) produces the same effect ; the index
scan is slower, but the heap scan uses the same amount of IO.

Two indexes, (order_date) and (product_id), strangely, do not produce a
BitmapAnd ; instead a plan with more IO is chosen.

>> - denormalization (ie adding a column in one of your tables and a
>> multicolumn index)
>
> For this query it is possible to duplicate kuupaev column to rid table.
> However most of the this seems to go to scanning rid table, so I suspect
> that this will help.

Yes, most of the time goes to scanning rid table, and this is the time
that should be reduced.
Adding a date column in "rid" would allow you to create a multicolumn
index on rid (dokumnr,date) which would massively speed up the particular
query above.
If you don't create a multicolumn index, this denormalization is useless.

Basically instead of scanning all rows in "rid" where

>
>> - materialized views
>> - materialized summary tables (ie. summary of sales for last month, for
>> instance)
>
> There are about 1000 items and reports are different.

It all depends on what you put in your summary table...

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T09:57:37Z

> How to vacuum full pg_shdepend automatically so that other users can
> work at same time ?

Your table is horribly bloated.
You must use VACUUM FULL + REINDEX (as superuser) on it, however
unfortunately, it is blocking.
Therefore, you should wait for sunday night to do this, when noone will
notice.
Meanwhile, you can always VACUUM it (as superuser) and REINDEX it.
And while you're at it, VACUUM FULL + reindex the entire database.

To avoid such annoyances in the future, you should ensure that autovacuum
runs properly ; you should investigate this. If you use a cron'ed VACUUM
that does not run as superuser, then it will not be able to VACUUM the
system catalogs, and the problem will come back.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T09:51:01Z

Richard,

Thank you.

> Try "SELECT count(*) FROM pg_shdepend".

This query returns 3625 and takes 35 seconds to run.

> If it's not a million rows, then the table is bloated. Try (as postgres
> or some other db superuser) "vacuum full pg_shdepend" and a "reindex
> pg_shdepend".

vacuum full verbose pg_shdepend
INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 16103561 removable, 3629 nonremovable row
versions in 131425 pages
DETAIL: 0 dead row versions cannot be removed yet.
Nonremovable row versions range from 49 to 49 bytes long.
There were 0 unused item pointers.
Total free space (including removable row versions) is 1009387632 bytes.
131363 pages are or will become empty, including 0 at the end of the table.
131425 pages containing 1009387632 free bytes are potential move
destinations.
CPU 2.12s/1.69u sec elapsed 52.66 sec.
INFO: index "pg_shdepend_depender_index" now contains 3629 row versions in
101794 pages
DETAIL: 16103561 index row versions were removed.
101311 index pages have been deleted, 20000 are currently reusable.
CPU 20.12s/14.52u sec elapsed 220.66 sec.

After 400 seconds of run I got phone calls that server does not respond to
other clients. So I was forced to cancel "vacuum full verbose pg_shdepend
" command.

How to run it so that other users can use database at same time ?

> If it is a million rows, you'll need to find out why. Do you have a lot
> of temporary tables that aren't being dropped or something similar?

Application creates temporary tables in many places. Every sales operation
probably creates some temporary tables.
Should I change something in configuration or change application (Only
single POS application which is used to access this db) or is only solution
to manully run

vacuum full pg_shdepend
reindex pg_shdepend

periodically ?
How to vacuum full pg_shdepend automatically so that other users can work at
same time ?

Hopefully this table size does not affect to query speed.

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T08:32:50Z

Andrus wrote:
>> - what's the size of the dataset relative to the RAM ?
>
> Db size is 7417 MB
> relevant table sizes in desc by size order:
>
> 1 40595 dok 2345 MB

> 2 1214 pg_shdepend 2259 MB
> 6 1232 pg_shdepend_depender_index 795 MB
> 7 1233 pg_shdepend_reference_index 438 MB

These three are highly suspicious. They track dependencies between
system object (so you can't drop function F because trigger T depends on
it).

http://www.postgresql.org/docs/8.3/static/catalog-pg-shdepend.html

You've got 3.5GB of data there, which is a *lot* of dependencies.

Try "SELECT count(*) FROM pg_shdepend".

If it's not a million rows, then the table is bloated. Try (as postgres
or some other db superuser) "vacuum full pg_shdepend" and a "reindex
pg_shdepend".

If it is a million rows, you'll need to find out why. Do you have a lot
of temporary tables that aren't being dropped or something similar?

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T08:15:10Z

Just the most important points:

1) "dok" table contains 1235086 row versions in 171641 pages (with 8kB
pages this means 1.4GB MB of data), but there are 1834279 unused item
pointers (i.e. about 60% of the space is wasted)

2) "rid" table contains 3275189 roiws in 165282 (with 8kB pages this means
about 1.3GB of data), but there are 1878923 unused item pointers (i.e.
about 30% of the space is wasted)

3) don't forget to execute analyze after vacuuming (or vacuum analyze)

4) I'm not sure why the sizes reported by you (for example 2.3GB vs 1.5GB
for "doc" table) - the difference seems too large for me.

Anyway the amount of wasted rows seems significant to me - I'd try to
solve this first. Either by VACUUM FULL or by CLUSTER. The CLUSTER will
lock the table exclusively, but the results may be better (when sorting by
a well chosen index). Don't forget to run ANALYZE afterwards.

Several other things to consider:

1) Regarding the toode column - why are you using CHAR(20) when the values
are actually shorter? This may significantly increase the amount of space
required.

2) I've noticed the CPU used is Celeron, which may negatively affect the
speed of hash computation. I'd try to replace it by something faster - say
INTEGER as an artificial primary key of the "toode" table and using it as
a FK in other tables. This might improve the "Bitmap Heap Scan on rid"
part, but yes - it's just a minor improvement compared to the "Hash Join"
part of the query.

Materialized views seem like a good idea to me, but maybe I'm not seeing
something. What do you mean by "reports are different"? If there is a lot
of rows for a given product / day, then creating an aggregated table with
(product code / day) as a primary key is quite simple. It may require a
lot of disk space, but it'll remove the hash join overhead. But if the
queries are very different, then it may be difficult to build such
materialized view(s).

regards
Tomas

> PFC,
>
> thank you.
>
>> OK so vmstat says you are IO-bound, this seems logical if the same plan
>> has widely varying timings...
>>
>> Let's look at the usual suspects :
>>
>> - how many dead rows in your tables ? are your tables data, or bloat ?
>> (check vacuum verbose, etc)
>
> set search_path to firma2,public;
> vacuum verbose dok; vacuum verbose rid
>
> INFO: vacuuming "firma2.dok"
> INFO: index "dok_pkey" now contains 1235086 row versions in 9454 pages
> DETAIL: 100 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.16s/0.38u sec elapsed 0.77 sec.
> INFO: index "dok_dokumnr_idx" now contains 1235086 row versions in 9454
> pages
> DETAIL: 100 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.14s/0.40u sec elapsed 0.78 sec.
> INFO: index "dok_klient_idx" now contains 1235086 row versions in 18147
> pages
> DETAIL: 887 index row versions were removed.
> 3265 index pages have been deleted, 3033 are currently reusable.
> CPU 0.36s/0.46u sec elapsed 31.87 sec.
> INFO: index "dok_krdokumnr_idx" now contains 1235086 row versions in
> 11387
> pages
> DETAIL: 119436 index row versions were removed.
> 1716 index pages have been deleted, 1582 are currently reusable.
> CPU 0.47s/0.55u sec elapsed 63.38 sec.
> INFO: index "dok_kuupaev_idx" now contains 1235101 row versions in 10766
> pages
> DETAIL: 119436 index row versions were removed.
> 659 index pages have been deleted, 625 are currently reusable.
> CPU 0.62s/0.53u sec elapsed 40.20 sec.
> INFO: index "dok_tasudok_idx" now contains 1235104 row versions in 31348
> pages
> DETAIL: 119436 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 1.18s/1.08u sec elapsed 118.97 sec.
> INFO: index "dok_tasudok_unique_idx" now contains 99 row versions in 97
> pages
> DETAIL: 98 index row versions were removed.
> 80 index pages have been deleted, 80 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.48 sec.
> INFO: index "dok_tasumata_idx" now contains 1235116 row versions in 11663
> pages
> DETAIL: 119436 index row versions were removed.
> 5340 index pages have been deleted, 5131 are currently reusable.
> CPU 0.43s/0.56u sec elapsed 53.96 sec.
> INFO: index "dok_tellimus_idx" now contains 1235122 row versions in 11442
> pages
> DETAIL: 119436 index row versions were removed.
> 1704 index pages have been deleted, 1569 are currently reusable.
> CPU 0.45s/0.59u sec elapsed 76.91 sec.
> INFO: index "dok_yksus_pattern_idx" now contains 1235143 row versions in
> 5549 pages
> DETAIL: 119436 index row versions were removed.
> 529 index pages have been deleted, 129 are currently reusable.
> CPU 0.19s/0.46u sec elapsed 2.72 sec.
> INFO: index "dok_doktyyp" now contains 1235143 row versions in 3899 pages
> DETAIL: 119436 index row versions were removed.
> 188 index pages have been deleted, 13 are currently reusable.
> CPU 0.14s/0.44u sec elapsed 1.40 sec.
> INFO: index "dok_sihtyksus_pattern_idx" now contains 1235143 row versions
> in 5353 pages
> DETAIL: 119436 index row versions were removed.
> 286 index pages have been deleted, 5 are currently reusable.
> CPU 0.13s/0.45u sec elapsed 3.01 sec.
> INFO: "dok": removed 119436 row versions in 13707 pages
> DETAIL: CPU 0.80s/0.37u sec elapsed 14.15 sec.
> INFO: "dok": found 119436 removable, 1235085 nonremovable row versions in
> 171641 pages
> DETAIL: 2 dead row versions cannot be removed yet.
> There were 1834279 unused item pointers.
> 0 pages are entirely empty.
> CPU 6.56s/6.88u sec elapsed 450.54 sec.
> INFO: vacuuming "pg_toast.pg_toast_40595"
> INFO: index "pg_toast_40595_index" now contains 0 row versions in 1 pages
> DETAIL: 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: "pg_toast_40595": found 0 removable, 0 nonremovable row versions in
> 0
> pages
> DETAIL: 0 dead row versions cannot be removed yet.
> There were 0 unused item pointers.
> 0 pages are entirely empty.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: vacuuming "firma2.rid"
> INFO: index "rid_pkey" now contains 3275197 row versions in 13959 pages
> DETAIL: 38331 index row versions were removed.
> 262 index pages have been deleted, 262 are currently reusable.
> CPU 0.42s/1.05u sec elapsed 58.56 sec.
> INFO: index "rid_dokumnr_idx" now contains 3275200 row versions in 14125
> pages
> DETAIL: 38331 index row versions were removed.
> 572 index pages have been deleted, 571 are currently reusable.
> CPU 0.49s/1.14u sec elapsed 71.57 sec.
> INFO: index "rid_inpdokumnr_idx" now contains 3275200 row versions in
> 15103
> pages
> DETAIL: 38331 index row versions were removed.
> 579 index pages have been deleted, 579 are currently reusable.
> CPU 0.66s/1.03u sec elapsed 68.38 sec.
> INFO: index "rid_toode_idx" now contains 3275224 row versions in 31094
> pages
> DETAIL: 38331 index row versions were removed.
> 2290 index pages have been deleted, 2290 are currently reusable.
> CPU 1.39s/1.58u sec elapsed 333.82 sec.
> INFO: index "rid_rtellimus_idx" now contains 3275230 row versions in 7390
> pages
> DETAIL: 18591 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.18s/0.66u sec elapsed 1.78 sec.
> INFO: index "rid_toode_pattern_idx" now contains 3275230 row versions in
> 16310 pages
> DETAIL: 17800 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.44s/1.04u sec elapsed 6.55 sec.
> INFO: "rid": removed 38331 row versions in 3090 pages
> DETAIL: CPU 0.20s/0.10u sec elapsed 5.49 sec.
> INFO: "rid": found 38331 removable, 3275189 nonremovable row versions in
> 165282 pages
> DETAIL: 0 dead row versions cannot be removed yet.
> There were 1878923 unused item pointers.
> 0 pages are entirely empty.
> CPU 5.06s/7.27u sec elapsed 607.59 sec.
>
> Query returned successfully with no result in 1058319 ms.
>
>> - what's the size of the dataset relative to the RAM ?
>
> Db size is 7417 MB
> relevant table sizes in desc by size order:
>
> 1 40595 dok 2345 MB
> 2 1214 pg_shdepend 2259 MB
> 3 40926 rid 2057 MB
> 6 1232 pg_shdepend_depender_index 795 MB
> 7 1233 pg_shdepend_reference_index 438 MB
> 8 44286 dok_tasudok_idx 245 MB
> 9 44299 rid_toode_idx 243 MB
> 10 44283 dok_klient_idx 142 MB
> 11 19103791 rid_toode_pattern_idx 127 MB
> 14 44298 rid_inpdokumnr_idx 118 MB
> 15 44297 rid_dokumnr_idx 110 MB
> 16 43573 rid_pkey 109 MB
> 18 44288 dok_tasumata_idx 91 MB
> 19 44289 dok_tellimus_idx 89 MB
> 20 44284 dok_krdokumnr_idx 89 MB
> 21 44285 dok_kuupaev_idx 84 MB
> 23 43479 dok_pkey 74 MB
> 24 44282 dok_dokumnr_idx 74 MB
> 25 19076304 rid_rtellimus_idx 58 MB
> 26 18663923 dok_yksus_pattern_idx 43 MB
> 27 18801591 dok_sihtyksus_pattern_idx 42 MB
> 29 18774881 dok_doktyyp 30 MB
> 46 40967 toode 13 MB
>
> server is HP Proliant DL320 G3
> http://h18000.www1.hp.com/products/quickspecs/12169_ca/12169_ca.HTML
> CPU is 2.93Ghz Celeron 256kb cache.
>
> Server has 2 GB RAM.
> It has SATA RAID 0,1 integrated controller (1.5Gbps) and SAMSUNG HD160JJ
> mirrored disks.
>
>> Now let's look more closely at the query :
>>
>> explain analyze
>> SELECT sum(1)
>> FROM dok JOIN rid USING (dokumnr)
>> JOIN toode USING (toode)
>> LEFT JOIN artliik using(grupp,liik)
>> WHERE rid.toode='X05' AND dok.kuupaev>='2008-09-01'
>>
>>
>> I presume doing the query without artliik changes nothing to the
>> runtime,
>> yes ?
>
> Yes. After removing artkliik from join I got response times 65 and 50
> seconds, so this does not make difference.
>
>> Your problem here is that, no matter what, postgres will have to examine
>> - all rows where dok.kuupaev>='2008-09-01',
>> - and all rows where rid.toode = 'X05'.
>> If you use dok.kuupaev>='2007-09-01' (note : 2007) it will probably have
>> to scan many, many more rows.
>
> Probably yes, since then it reads one year more sales data.
>
>> If you perform this query often you could CLUSTER rid on (toode) and dok
>> on (kuupaev), but this can screw other queries.
>
> Some reports are by sales date (dok.kuupaev) and customers.
> CLUSTER rid on (toode) slows them down. Also autovacuum cannot do
> clustering.
>
>> What is the meaning of the columns ?
>
> This is typical sales data:
>
> -- Receipt headers:
> DOK ( dokumnr INT SERIAL PRIMARY KEY,
> kuupaev DATE --- sales date
> )
>
> -- Receipt details
> RID ( dokumnr INT,
> toode CHAR(20), -- item code
> CONSTRAINT rid_dokumnr_fkey FOREIGN KEY (dokumnr) REFERENCES dok
> (dokumnr),
> CONSTRAINT rid_toode_fkey FOREIGN KEY (toode)
> REFERENCES firma2.toode (toode)
> )
>
> -- Products
> TOODE (
> toode CHAR(20) PRIMARY KEY
> )
>
>> To make this type of query faster I would tend to think about :
>
>> - denormalization (ie adding a column in one of your tables and a
>> multicolumn index)
>
> For this query it is possible to duplicate kuupaev column to rid table.
> However most of the this seems to go to scanning rid table, so I suspect
> that this will help.
>
>> - materialized views
>> - materialized summary tables (ie. summary of sales for last month, for
>> instance)
>
> There are about 1000 items and reports are different.
>
> Andrus.
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@...)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T07:12:27Z

PFC,

thank you.

> OK so vmstat says you are IO-bound, this seems logical if the same plan
> has widely varying timings...
>
> Let's look at the usual suspects :
>
> - how many dead rows in your tables ? are your tables data, or bloat ?
> (check vacuum verbose, etc)

set search_path to firma2,public;
vacuum verbose dok; vacuum verbose rid

INFO: vacuuming "firma2.dok"
INFO: index "dok_pkey" now contains 1235086 row versions in 9454 pages
DETAIL: 100 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.16s/0.38u sec elapsed 0.77 sec.
INFO: index "dok_dokumnr_idx" now contains 1235086 row versions in 9454
pages
DETAIL: 100 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.14s/0.40u sec elapsed 0.78 sec.
INFO: index "dok_klient_idx" now contains 1235086 row versions in 18147
pages
DETAIL: 887 index row versions were removed.
3265 index pages have been deleted, 3033 are currently reusable.
CPU 0.36s/0.46u sec elapsed 31.87 sec.
INFO: index "dok_krdokumnr_idx" now contains 1235086 row versions in 11387
pages
DETAIL: 119436 index row versions were removed.
1716 index pages have been deleted, 1582 are currently reusable.
CPU 0.47s/0.55u sec elapsed 63.38 sec.
INFO: index "dok_kuupaev_idx" now contains 1235101 row versions in 10766
pages
DETAIL: 119436 index row versions were removed.
659 index pages have been deleted, 625 are currently reusable.
CPU 0.62s/0.53u sec elapsed 40.20 sec.
INFO: index "dok_tasudok_idx" now contains 1235104 row versions in 31348
pages
DETAIL: 119436 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 1.18s/1.08u sec elapsed 118.97 sec.
INFO: index "dok_tasudok_unique_idx" now contains 99 row versions in 97
pages
DETAIL: 98 index row versions were removed.
80 index pages have been deleted, 80 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.48 sec.
INFO: index "dok_tasumata_idx" now contains 1235116 row versions in 11663
pages
DETAIL: 119436 index row versions were removed.
5340 index pages have been deleted, 5131 are currently reusable.
CPU 0.43s/0.56u sec elapsed 53.96 sec.
INFO: index "dok_tellimus_idx" now contains 1235122 row versions in 11442
pages
DETAIL: 119436 index row versions were removed.
1704 index pages have been deleted, 1569 are currently reusable.
CPU 0.45s/0.59u sec elapsed 76.91 sec.
INFO: index "dok_yksus_pattern_idx" now contains 1235143 row versions in
5549 pages
DETAIL: 119436 index row versions were removed.
529 index pages have been deleted, 129 are currently reusable.
CPU 0.19s/0.46u sec elapsed 2.72 sec.
INFO: index "dok_doktyyp" now contains 1235143 row versions in 3899 pages
DETAIL: 119436 index row versions were removed.
188 index pages have been deleted, 13 are currently reusable.
CPU 0.14s/0.44u sec elapsed 1.40 sec.
INFO: index "dok_sihtyksus_pattern_idx" now contains 1235143 row versions
in 5353 pages
DETAIL: 119436 index row versions were removed.
286 index pages have been deleted, 5 are currently reusable.
CPU 0.13s/0.45u sec elapsed 3.01 sec.
INFO: "dok": removed 119436 row versions in 13707 pages
DETAIL: CPU 0.80s/0.37u sec elapsed 14.15 sec.
INFO: "dok": found 119436 removable, 1235085 nonremovable row versions in
171641 pages
DETAIL: 2 dead row versions cannot be removed yet.
There were 1834279 unused item pointers.
0 pages are entirely empty.
CPU 6.56s/6.88u sec elapsed 450.54 sec.
INFO: vacuuming "pg_toast.pg_toast_40595"
INFO: index "pg_toast_40595_index" now contains 0 row versions in 1 pages
DETAIL: 0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: "pg_toast_40595": found 0 removable, 0 nonremovable row versions in 0
pages
DETAIL: 0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: vacuuming "firma2.rid"
INFO: index "rid_pkey" now contains 3275197 row versions in 13959 pages
DETAIL: 38331 index row versions were removed.
262 index pages have been deleted, 262 are currently reusable.
CPU 0.42s/1.05u sec elapsed 58.56 sec.
INFO: index "rid_dokumnr_idx" now contains 3275200 row versions in 14125
pages
DETAIL: 38331 index row versions were removed.
572 index pages have been deleted, 571 are currently reusable.
CPU 0.49s/1.14u sec elapsed 71.57 sec.
INFO: index "rid_inpdokumnr_idx" now contains 3275200 row versions in 15103
pages
DETAIL: 38331 index row versions were removed.
579 index pages have been deleted, 579 are currently reusable.
CPU 0.66s/1.03u sec elapsed 68.38 sec.
INFO: index "rid_toode_idx" now contains 3275224 row versions in 31094
pages
DETAIL: 38331 index row versions were removed.
2290 index pages have been deleted, 2290 are currently reusable.
CPU 1.39s/1.58u sec elapsed 333.82 sec.
INFO: index "rid_rtellimus_idx" now contains 3275230 row versions in 7390
pages
DETAIL: 18591 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.18s/0.66u sec elapsed 1.78 sec.
INFO: index "rid_toode_pattern_idx" now contains 3275230 row versions in
16310 pages
DETAIL: 17800 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.44s/1.04u sec elapsed 6.55 sec.
INFO: "rid": removed 38331 row versions in 3090 pages
DETAIL: CPU 0.20s/0.10u sec elapsed 5.49 sec.
INFO: "rid": found 38331 removable, 3275189 nonremovable row versions in
165282 pages
DETAIL: 0 dead row versions cannot be removed yet.
There were 1878923 unused item pointers.
0 pages are entirely empty.
CPU 5.06s/7.27u sec elapsed 607.59 sec.

Query returned successfully with no result in 1058319 ms.

> - what's the size of the dataset relative to the RAM ?

Db size is 7417 MB
relevant table sizes in desc by size order:

1 40595 dok 2345 MB
2 1214 pg_shdepend 2259 MB
3 40926 rid 2057 MB
6 1232 pg_shdepend_depender_index 795 MB
7 1233 pg_shdepend_reference_index 438 MB
8 44286 dok_tasudok_idx 245 MB
9 44299 rid_toode_idx 243 MB
10 44283 dok_klient_idx 142 MB
11 19103791 rid_toode_pattern_idx 127 MB
14 44298 rid_inpdokumnr_idx 118 MB
15 44297 rid_dokumnr_idx 110 MB
16 43573 rid_pkey 109 MB
18 44288 dok_tasumata_idx 91 MB
19 44289 dok_tellimus_idx 89 MB
20 44284 dok_krdokumnr_idx 89 MB
21 44285 dok_kuupaev_idx 84 MB
23 43479 dok_pkey 74 MB
24 44282 dok_dokumnr_idx 74 MB
25 19076304 rid_rtellimus_idx 58 MB
26 18663923 dok_yksus_pattern_idx 43 MB
27 18801591 dok_sihtyksus_pattern_idx 42 MB
29 18774881 dok_doktyyp 30 MB
46 40967 toode 13 MB

server is HP Proliant DL320 G3
http://h18000.www1.hp.com/products/quickspecs/12169_ca/12169_ca.HTML
CPU is 2.93Ghz Celeron 256kb cache.

Server has 2 GB RAM.
It has SATA RAID 0,1 integrated controller (1.5Gbps) and SAMSUNG HD160JJ
mirrored disks.

> Now let's look more closely at the query :
>
> explain analyze
> SELECT sum(1)
> FROM dok JOIN rid USING (dokumnr)
> JOIN toode USING (toode)
> LEFT JOIN artliik using(grupp,liik)
> WHERE rid.toode='X05' AND dok.kuupaev>='2008-09-01'
>
>
> I presume doing the query without artliik changes nothing to the runtime,
> yes ?

Yes. After removing artkliik from join I got response times 65 and 50
seconds, so this does not make difference.

> Your problem here is that, no matter what, postgres will have to examine
> - all rows where dok.kuupaev>='2008-09-01',
> - and all rows where rid.toode = 'X05'.
> If you use dok.kuupaev>='2007-09-01' (note : 2007) it will probably have
> to scan many, many more rows.

Probably yes, since then it reads one year more sales data.

> If you perform this query often you could CLUSTER rid on (toode) and dok
> on (kuupaev), but this can screw other queries.

Some reports are by sales date (dok.kuupaev) and customers.
CLUSTER rid on (toode) slows them down. Also autovacuum cannot do
clustering.

> What is the meaning of the columns ?

This is typical sales data:

-- Receipt headers:
DOK ( dokumnr INT SERIAL PRIMARY KEY,
kuupaev DATE --- sales date
)

-- Receipt details
RID ( dokumnr INT,
toode CHAR(20), -- item code
CONSTRAINT rid_dokumnr_fkey FOREIGN KEY (dokumnr) REFERENCES dok
(dokumnr),
CONSTRAINT rid_toode_fkey FOREIGN KEY (toode)
REFERENCES firma2.toode (toode)
)

-- Products
TOODE (
toode CHAR(20) PRIMARY KEY
)

> To make this type of query faster I would tend to think about :

> - denormalization (ie adding a column in one of your tables and a
> multicolumn index)

For this query it is possible to duplicate kuupaev column to rid table.
However most of the this seems to go to scanning rid table, so I suspect
that this will help.

> - materialized views
> - materialized summary tables (ie. summary of sales for last month, for
> instance)

There are about 1000 items and reports are different.

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Hash join on int takes 8..114 seconds

2008-11-21T05:49:24Z

Richard,

> In addition to "top" below, you'll probably find "vmstat 5" useful.

Thank you.
During this query run (65 sec), vmstat 5 shows big values in bi,cs and wa
columns:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
1 1 88 51444 0 1854404 0 0 17 6 15 13 5 1 83
10
0 1 88 52140 0 1854304 0 0 3626 95 562 784 15 38 0
47
0 1 92 51608 0 1855668 0 0 14116 103 1382 2294 4 8 0
88
0 1 92 51620 0 1857256 0 1 15258 31 1210 1975 4 8 0
88
0 2 92 50448 0 1859188 0 0 13118 19 1227 1982 3 7 0
90
0 1 92 51272 0 1859088 0 0 7691 53 828 1284 14 4 0
82
0 1 92 52472 0 1858792 0 0 10691 9 758 968 3 7 0
89
0 2 92 51240 0 1858596 0 0 8204 7407 717 1063 2 5 0
93
0 1 92 51648 0 1860388 0 0 20622 121 1118 2229 12 9 0
79
2 1 92 50564 0 1861396 0 0 20994 3277 969 1681 15 8 0
76
1 0 92 52180 0 1860192 0 0 18542 36 802 1276 36 12 0
51
0 0 92 91872 0 1820948 0 1 15285 47 588 774 9 12 32
47
0 0 92 91904 0 1820948 0 0 0 4 251 18 0 0
100 0
0 0 92 92044 0 1820948 0 0 0 0 250 17 0 0
100 0
0 0 92 91668 0 1821156 0 0 27 93 272 66 5 0 92
3
0 0 92 91668 0 1821156 0 0 0 64 260 38 0 0
100 0
0 0 92 91636 0 1821156 0 0 0 226 277 71 0 0
100 0
0 0 92 91676 0 1821156 0 0 0 26 255 23 0 0
100 0

> Here you're stuck waiting for disks (91.0% wa). Check out vmstat and
> iostat to see what's happening.

typing iostat returns

bash: iostat: command not found

It seems that this is not installed in this gentoo.

Andrus.

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: Performance and IN clauses

2008-11-20T17:09:36Z

Mark Roberts napsal(a):

> On Tue, 2008-11-18 at 17:38 +0100, tv@... wrote:
>> I bet there is no 'critical' length - this is just another case of
>> index
>> scan vs. seqscan. The efficiency depends on the size of the table /
>> row,
>> amount of data in the table, variability of the column used in the IN
>> clause, etc.
>>
>> Splitting the query with 1000 items into 10 separate queries, the
>> smaller
>> queries may be faster but the total time consumed may be actually
>> higher.
>> Something like
>>
>> 10 * (time of small query) + (time to combine them) > (time of large
>> query)
>>
>> If the performance of the 'split' solution is actually better than the
>> original query, it just means that the planner does not use index scan
>> when it actually should. That means that either
>>
>> (a) the planner is not smart enough
>> (b) it has not current statistics of the table (run ANALYZE on the
>> table)
>> (c) the statistics are not detailed enough (ALTER TABLE ... SET
>> STATICTICS)
>> (d) the cost variables are not set properly (do not match the hardware
>> -
>> decreate index scan cost / increase seq scan cost)
>>
>> regards
>> Tomas
>
> I know that it's much faster (for us) to run many smaller queries than
> one large query, and I think that it's primarily because of your reason
> a. Most of our problems come from Pg misunderstanding the results of a
> join and making a bad plan decision. Batching dramatically reduces the
> liklihood of this.

As I already said - even the smartest planner won't work without correct
input data. Have you tried fixing the points (b), (c) and (d)?

Fixing them might improve the planner performance so that you don't need
the batchning at all.

regards
Tomas

--
Sent via pgsql-performance mailing list (pgsql-performance@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance