38761 Commits

Author SHA1 Message Date
Tom Lane
549d2a238a Allow UNLISTEN in hot-standby mode.
Since LISTEN is (still) disallowed, UNLISTEN must be a no-op in a
hot-standby session, and so there's no harm in allowing it.  This
change allows client code to not worry about whether it's connected
to a primary or standby server when performing session-state-reset
type activities.  (Note that DISCARD ALL, which includes UNLISTEN,
was already allowed, making it inconsistent to reject UNLISTEN.)

Per discussion, back-patch to all supported versions.

Shay Rojansky, reviewed by Mi Tar

Discussion: https://postgr.es/m/CADT4RqCf2gA_TJtPAjnGzkC3ZiexfBZiLmA-mV66e4UyuVv8bA@mail.gmail.com
2019-01-25 21:14:31 -05:00
Tom Lane
6d6333745b Remove infinite-loop hazards in ecpg test suite.
A report from Andrew Dunstan showed that an ecpglib breakage that
causes repeated query failures could lead to infinite loops in some
ecpg test scripts, because they contain "while(1)" loops with no
exit condition other than successful test completion.  That might
be all right for manual testing, but it seems entirely unacceptable
for automated test environments such as our buildfarm.  We don't
want buildfarm owners to have to intervene manually when a test
goes wrong.

To fix, just change all those while(1) loops to exit after at most
100 iterations (which is more than any of them expect to iterate).
This seems sufficient since we'd see discrepancies in the test output
if any loop executed the wrong number of times.

I tested this by dint of intentionally breaking ecpg_do_prologue
to always fail, and verifying that the tests still got to completion.

Back-patch to all supported branches, since the whole point of this
exercise is to protect the buildfarm against future mistakes.

Discussion: https://postgr.es/m/18693.1548302004@sss.pgh.pa.us
2019-01-24 16:46:56 -05:00
Tom Lane
773f594408 Blind attempt to fix _configthreadlocale() failures on MinGW.
Apparently, some builds of MinGW contain a version of
_configthreadlocale() that always returns -1, indicating failure.
Rather than treating that as a curl-up-and-die condition, soldier on
as though the function didn't exist.  This leaves us without thread
safety on such MinGW versions, but we didn't have it anyway.

Discussion: https://postgr.es/m/d06a16bc-52d6-9f0d-2379-21242d7dbe81@2ndQuadrant.com
2019-01-23 22:47:23 -05:00
Heikki Linnakangas
e46a475718 Fix misc typos in comments.
Spotted mostly by Fabien Coelho.

Discussion: https://www.postgresql.org/message-id/alpine.DEB.2.21.1901230947050.16643@lancre
2019-01-23 13:47:05 +02:00
Tom Lane
c9ff6cbfc4 Avoid thread-safety problem in ecpglib.
ecpglib attempts to force the LC_NUMERIC locale to "C" while reading
server output, to avoid problems with strtod() and related functions.
Historically it's just issued setlocale() calls to do that, but that
has major problems if we're in a threaded application.  setlocale()
itself is not required by POSIX to be thread-safe (and indeed is not,
on recent OpenBSD).  Moreover, its effects are process-wide, so that
we could cause unexpected results in other threads, or another thread
could change our setting.

On platforms having uselocale(), which is required by POSIX:2008,
we can avoid these problems by using uselocale() instead.  Windows
goes its own way as usual, but we can make it safe by using
_configthreadlocale().  Platforms having neither continue to use the
old code, but that should be pretty much nobody among current systems.

(Subsequent buildfarm results show that recent NetBSD versions still
lack uselocale(), but it's not a big problem because they also do not
support non-"C" settings for LC_NUMERIC.)

Back-patch of commits 8eb4a9312 and ee27584c4.

Michael Meskes and Tom Lane; thanks also to Takayuki Tsunakawa.

Discussion: https://postgr.es/m/31420.1547783697@sss.pgh.pa.us
2019-01-21 23:18:58 -05:00
Tomas Vondra
6f090c1d8d Revert "Add valgrind suppressions for wcsrtombs optimizations"
This reverts commit 41344896364c4bf2229ec590c95cf23a6bec928e.

Per discussion, it's not desirable to add valgrind suppressions for
outside our own code base (e.g. glibc in this case), especially when
the suppressions may be platform-specific. There are better ways to
deal with that, e.g. by providing local suppressions.

Discussion: https://www.postgresql.org/message-id/flat/90ac0452-e907-e7a4-b3c8-15bd33780e62%402ndquadrant.com
2019-01-19 20:36:07 +01:00
Tom Lane
b161ffe317 Use our own getopt() on OpenBSD.
Recent OpenBSD (at least 5.9 and up) has a version of getopt(3)
that will not cope with the "-:" spec we use to accept double-dash
options in postgres.c and postmaster.c.  Admittedly, that's a hack
because POSIX only requires getopt() to allow alphanumeric option
characters.  I have no desire to find another way, however, so
let's just do what we were already doing on Solaris: force use
of our own src/port/getopt.c implementation.

In passing, improve some of the comments around said implementation.

Per buildfarm and local testing.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/30197.1547835700@sss.pgh.pa.us
2019-01-18 15:06:26 -05:00
Magnus Hagander
1c57acc655 Replace references to mailinglists with @lists.postgresql.org
The namespace for all lists have changed a while ago, so all references
should use the correct address.
2019-01-17 13:55:27 +01:00
Magnus Hagander
a3f632aad9 Remove references to Majordomo
Lists are not handled by Majordomo anymore and haven't been for
a while, so remove the reference and instead direct people to the
list server.
2019-01-17 13:55:25 +01:00
Andrew Gierth
174fab9934 Postpone aggregate checks until after collation is assigned.
Previously, parseCheckAggregates was run before
assign_query_collations, but this causes problems if any expression
has already had a collation assigned by some transform function (e.g.
transformCaseExpr) before parseCheckAggregates runs. The differing
collations would cause expressions not to be recognized as equal to
the ones in the GROUP BY clause, leading to spurious errors about
unaggregated column references.

The result was that CASE expr WHEN val ... would fail when "expr"
contained a GROUPING() expression or matched one of the group by
expressions, and where collatable types were involved; whereas the
supposedly identical CASE WHEN expr = val ... would succeed.

Backpatch all the way; this appears to have been wrong ever since
collations were introduced.

Per report from Guillaume Lelarge, analysis and patch by me.

Discussion: https://postgr.es/m/CAECtzeVSO_US8C2Khgfv54ZMUOBR4sWq+6_bLrETnWExHT=rFg@mail.gmail.com
Discussion: https://postgr.es/m/87muo0k0c7.fsf@news-spur.riddles.org.uk
2019-01-17 06:35:31 +00:00
Andrew Dunstan
feea3858f0 fix typo 2019-01-13 16:44:31 -05:00
Andrew Dunstan
4ef6a2c78b Make DLSUFFIX easily discoverable by build scripts
This will enable things like the buildfarm client to discover more
reliably if certain libraries have been installed.

Discussion: https://postgr.es/m/859e7c91-7ef4-d4b4-2ca2-8046e0cbee09@2ndQuadrant.com

Backpatch to all live branches.
2019-01-13 16:06:28 -05:00
Tom Lane
8fd909436a Doc: update our docs about kernel IPC parameters on *BSD.
runtime.sgml said that you couldn't change SysV IPC parameters on OpenBSD
except by rebuilding the kernel.  That's definitely wrong in OpenBSD 6.x,
and excavation in their man pages says it changed in OpenBSD 3.3.

Update NetBSD and OpenBSD sections to recommend adjustment of the SEMMNI
and SEMMNS settings, which are painfully small by default on those
platforms.  (The discussion thread contemplated recommending that
people select POSIX semaphores instead, but the performance consequences
of that aren't really clear, so I'll refrain.)

Remove pointless discussion of SEMMNU and SEMMAP from the FreeBSD
section.  Minor other wordsmithing.

Discussion: https://postgr.es/m/27582.1546928073@sss.pgh.pa.us
2019-01-08 12:03:54 -05:00
Andrew Gierth
eef6476b2e doc: document that INFO messages always go to client.
In passing add a couple of links to the message severity table.

Backpatch because it's always been this way.

Author: Karl O. Pinc <kop@meme.com>
2019-01-07 18:57:10 +00:00
Michael Paquier
1d9dfc77be doc: Update RFC URLs
Consistently use the IETF HTML links instead of a random mix of
different sites and formats.  This also fixes one broken link for JSON
documentation.
2019-01-05 12:49:58 +09:00
Tom Lane
3010de813f Improve ANALYZE's handling of concurrent-update scenarios.
This patch changes the rule for whether or not a tuple seen by ANALYZE
should be included in its sample.

When we last touched this logic, in commit 51e1445f1, we weren't
thinking very hard about tuples being UPDATEd by a long-running
concurrent transaction.  In such a case, we might see the pre-image as
either LIVE or DELETE_IN_PROGRESS depending on timing; and we might see
the post-image not at all, or as INSERT_IN_PROGRESS.  Since the existing
code will not sample either DELETE_IN_PROGRESS or INSERT_IN_PROGRESS
tuples, this leads to concurrently-updated rows being omitted from the
sample entirely.  That's not very helpful, and it's especially the wrong
thing if the concurrent transaction ends up rolling back.

The right thing seems to be to sample DELETE_IN_PROGRESS rows just as if
they were live.  This makes the "sample it" and "count it" decisions the
same, which seems good for consistency.  It's clearly the right thing
if the concurrent transaction ends up rolling back; in effect, we are
sampling as though IN_PROGRESS transactions haven't happened yet.
Also, this combination of choices ensures maximum robustness against
the different combinations of whether and in which state we might see the
pre- and post-images of an update.

It's slightly annoying that we end up recording immediately-out-of-date
stats in the case where the transaction does commit, but on the other
hand the stats are fine for columns that didn't change in the update.
And the alternative of sampling INSERT_IN_PROGRESS rows instead seems
like a bad idea, because then the sampling would be inconsistent with
the way rows are counted for the stats report.

Per report from Mark Chambers; thanks to Jeff Janes for diagnosing
what was happening.  Back-patch to all supported versions.

Discussion: https://postgr.es/m/CAFh58O_Myr6G3tcH3gcGrF-=OExB08PJdWZcSBcEcovaiPsrHA@mail.gmail.com
2019-01-03 17:00:08 -05:00
Tom Lane
d6b37cdb6e Don't believe MinMaxExpr is leakproof without checking.
MinMaxExpr invokes the btree comparison function for its input datatype,
so it's only leakproof if that function is.  Many such functions are
indeed leakproof, but others are not, and we should not just assume that
they are.  Hence, adjust contain_leaked_vars to verify the leakproofness
of the referenced function explicitly.

I didn't add a regression test because it would need to depend on
some particular comparison function being leaky, and that's a moving
target, per discussion.

This has been wrong all along, so back-patch to supported branches.

Discussion: https://postgr.es/m/31042.1546194242@sss.pgh.pa.us
2019-01-02 16:33:48 -05:00
Bruce Momjian
dcb04c44af Update copyright for 2019
Backpatch-through: certain files through 9.4
2019-01-02 12:44:25 -05:00
Noah Misch
309d16f073 pg_regress: Promptly detect failed postmaster startup.
Detect it the way pg_ctl's wait_for_postmaster() does.  When pg_regress
spawned a postmaster that failed startup, we were detecting that only
with "pg_regress: postmaster did not respond within 60 seconds".
Back-patch to 9.4 (all supported versions).

Reviewed by Tom Lane.

Discussion: https://postgr.es/m/20181231172922.GA199150@gust.leadboat.com
2018-12-31 13:51:30 -08:00
Alvaro Herrera
2602838fa3 Have DISCARD ALL/TEMP remove leftover temp tables
Previously, it would only remove temp tables created in the same
session; but if the session uses the BackendId of a previously crashed
backend that left temp tables around, those would not get removed.
Since autovacuum would not drop them either (because it sees that the
BackendId is in use by the current session) these can cause annoying
xid-wraparound warnings.

Apply to branches 9.4 to 10.  This is not a problem since version 11,
because commit 943576bddcb5 added state tracking that makes autovacuum
realize that those temp tables are not ours, so it removes them.

This is useful to handle in DISCARD, because even though it does not
handle all situations, it does handle the common one where a connection
pooler keeps the same session open for an indefinitely long time.

Discussion: https://postgr.es/m/20181226190834.wsk2wzott5yzrjiq@alvherre.pgsql
Reviewed-by: Takayuki Tsunakawa, Michaël Paquier
2018-12-27 16:17:40 -03:00
Alvaro Herrera
5199abaca3 Make autovacuum more selective about temp tables to keep
When temp tables are in danger of XID wraparound, autovacuum drops them;
however, it preserves those that are owned by a working session.  This
is desirable, except when the session is connected to a different
database (because the temp tables cannot be from that session), so make
it only keep the temp tables only if the backend is in the same database
as the temp tables.

This is not bulletproof: it fails to detect temp tables left by a
session whose backend ID is reused in the same database but the new
session does not use temp tables.  Commit 943576bddcb5 fixes that case
too, for branches 11 and up (which is why we don't apply this fix to
those branches), but back-patching that one is not universally agreed
on.

Discussion: https://postgr.es/m/20181214162843.37g6h3txto43akrb@alvherre.pgsql
Reviewed-by: Takayuki Tsunakawa, Michaël Paquier
2018-12-27 16:01:36 -03:00
Michael Paquier
1d70076710 Ignore inherited temp relations from other sessions when truncating
Inheritance trees can include temporary tables if the parent is
permanent, which makes possible the presence of multiple temporary
children from different sessions.  Trying to issue a TRUNCATE on the
parent in this scenario causes a failure, so similarly to any other
queries just ignore such cases, which makes TRUNCATE work
transparently.

This makes truncation behave similarly to any other DML query working on
the parent table with queries which need to be issues on children.  A
set of isolation tests is added to cover basic cases.

Reported-by: Zhou Digoal
Author: Amit Langote, Michael Paquier
Discussion: https://postgr.es/m/15565-ce67a48d0244436a@postgresql.org
Backpatch-through: 9.4
2018-12-27 10:17:42 +09:00
Tom Lane
4f7ab73106 Fix portability failure introduced in commits d2b0b60e7 et al.
I made a frontend fprintf() format use %m, forgetting that that's only
safe in HEAD not the back branches; prior to 96bf88d52 and d6c55de1f,
it would work on glibc platforms but not elsewhere.  Revert to using
%s ... strerror(errno) as the code did before.

We could have left HEAD as-is, but for code consistency across branches,
I chose to apply this patch there too.

Per Coverity and a few buildfarm members.
2018-12-26 15:30:40 -05:00
Peter Eisentraut
b7b0314b80 Fix ancient compiler warnings and typos in !HAVE_SYMLINK code
This has never been correct since this code was introduced.
2018-12-22 07:27:21 +01:00
Tom Lane
773e2d29cd Doc: fix ancient mistake in search_path documentation.
"$user" in a search_path string is replaced by CURRENT_USER not
SESSION_USER.  (It actually was SESSION_USER in the initial implementation,
but we changed it shortly later, and evidently forgot to fix the docs to
match.)

Noted by antonov@stdpr.ru

Discussion: https://postgr.es/m/159151fb45d490c8d31ea9707e9ba99d@stdpr.ru
2018-12-20 13:55:11 -05:00
Tom Lane
ef673a32d3 Fix ancient thinko in mergejoin cost estimation.
"rescanratio" was computed as 1 + rescanned-tuples / total-inner-tuples,
which is sensible if it's to be multiplied by total-inner-tuples or a cost
value corresponding to scanning all the inner tuples.  But in reality it
was (mostly) multiplied by inner_rows or a related cost, numbers that take
into account the possibility of stopping short of scanning the whole inner
relation thanks to a limited key range in the outer relation.  This'd
still make sense if we could expect that stopping short would result in a
proportional decrease in the number of tuples that have to be rescanned.
It does not, however.  The argument that establishes the validity of our
estimate for that number is independent of whether we scan all of the inner
relation or stop short, and experimentation also shows that stopping short
doesn't reduce the number of rescanned tuples.  So the correct calculation
is 1 + rescanned-tuples / inner_rows, and we should be sure to multiply
that by inner_rows or a corresponding cost value.

Most of the time this doesn't make much difference, but if we have
both a high rescan rate (due to lots of duplicate values) and an outer
key range much smaller than the inner key range, then the error can
be significant, leading to a large underestimate of the cost associated
with rescanning.

Per report from Vijaykumar Jain.  This thinko appears to go all the way
back to the introduction of the rescan estimation logic in commit
70fba7043, so back-patch to all supported branches.

Discussion: https://postgr.es/m/CAE7uO5hMb_TZYJcZmLAgO6iD68AkEK6qCe7i=vZUkCpoKns+EQ@mail.gmail.com
2018-12-18 11:19:39 -05:00
Michael Paquier
a75c933a75 Update project link of pgBadger in documentation
The project has moved to a new place.

Reported-by: Peter Neave
Discussion: https://postgr.es/m/154474118231.5066.16352227860913505754@wrigleys.postgresql.org
2018-12-18 10:03:19 +09:00
Michael Paquier
696c68c2be Fix use-after-free bug when renaming constraints
This is an oversight from recent commit b13fd344.  While on it, tweak
the previous test with a better name for the renamed primary key.

Detected by buildfarm member prion which forces relation cache release
with -DRELCACHE_FORCE_RELEASE.  Back-patch down to 9.4 as the previous
commit.
2018-12-17 12:44:09 +09:00
Michael Paquier
d5d86e2cd6 Make constraint rename issue relcache invalidation on target relation
When a constraint gets renamed, it may have associated with it a target
relation (for example domain constraints don't have one).  Not
invalidating the target relation cache when issuing the renaming can
result in issues with subsequent commands that refer to the old
constraint name using the relation cache, causing various failures.  One
pattern spotted was using CREATE TABLE LIKE after a constraint
renaming.

Reported-by: Stuart <sfbarbee@gmail.com>
Author: Amit Langote
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/2047094.V130LYfLq4@station53.ousa.org
2018-12-17 10:37:24 +09:00
Tom Lane
b10d2252b3 Make error handling in parallel pg_upgrade less bogus.
reap_child() basically ignored the possibility of either an error in
waitpid() itself or a child process failure on signal.  We don't really
need to do more than report and crash hard, but proceeding as though
nothing is wrong is definitely Not Acceptable.  The error report for
nonzero child exit status was pretty off-point, as well.

Noted while fooling around with child-process failure detection
logic elsewhere.  It's been like this a long time, so back-patch to
all supported branches.
2018-12-16 14:51:48 -05:00
Alexander Korotkov
bf0e5a73be Fix wrong backpatching of ginRedoDeletePage() deadlock fix
19cf52e6cc changes lock order in ginRedoDeletePage().  But did it in a wrong
way due to oversight during backpatching.  This commit fixes that.

Reported-by: Bruce Momjian
Discussion: https://postgr.es/m/20181213153232.GA10664%40momjian.us
2018-12-13 22:32:05 +03:00
Alexander Korotkov
1cf175c74f Prevent GIN deleted pages from being reclaimed too early
When GIN vacuum deletes a posting tree page, it assumes that no concurrent
searchers can access it, thanks to ginStepRight() locking two pages at once.
However, since 9.4 searches can skip parts of posting trees descending from the
root.  That leads to the risk that page is deleted and reclaimed before
concurrent search can access it.

This commit prevents the risk of above by waiting for every transaction, which
might wait to reference this page, to finish.  Due to binary compatibility
we can't change GinPageOpaqueData to store corresponding transaction id.
Instead we reuse page header pd_prune_xid field, which is unused in index pages.

Discussion: https://postgr.es/m/31a702a.14dd.166c1366ac1.Coremail.chjischj%40163.com
Author: Andrey Borodin, Alexander Korotkov
Reviewed-by: Alexander Korotkov
Backpatch-through: 9.4
2018-12-13 06:52:26 +03:00
Alexander Korotkov
19cf52e6cc Prevent deadlock in ginRedoDeletePage()
On standby ginRedoDeletePage() can work concurrently with read-only queries.
Those queries can traverse posting tree in two ways.
1) Using rightlinks by ginStepRight(), which locks the next page before
   unlocking its left sibling.
2) Using downlinks by ginFindLeafPage(), which locks at most one page at time.

Original lock order was: page, parent, left sibling.  That lock order can
deadlock with ginStepRight().  In order to prevent deadlock this commit changes
lock order to: left sibling, page, parent.  Note, that position of parent in
locking order seems insignificant, because we only lock one page at time while
traversing downlinks.

Reported-by: Chen Huajun
Diagnosed-by: Chen Huajun, Peter Geoghegan, Andrey Borodin
Discussion: https://postgr.es/m/31a702a.14dd.166c1366ac1.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Backpatch-through: 9.4
2018-12-13 06:36:54 +03:00
Tom Lane
a96722938e Doc: improve documentation about ALTER LARGE OBJECT requirements.
Unlike other ALTER ref pages, this one neglected to mention that
ALTER OWNER requires being a member of the new owning role.
Per bug #15546 from Stefan Kadow.

Discussion: https://postgr.es/m/15546-0558c75fd2025e7c@postgresql.org
2018-12-11 11:22:28 -05:00
Tom Lane
3af726a1f3 Add stack depth checks to key recursive functions in backend/nodes/*.c.
Although copyfuncs.c has a check_stack_depth call in its recursion,
equalfuncs.c, outfuncs.c, and readfuncs.c lacked one.  This seems
unwise.

Likewise fix planstate_tree_walker(), in branches where that exists.

Discussion: https://postgr.es/m/30253.1544286631@sss.pgh.pa.us
2018-12-10 11:12:43 -05:00
Tom Lane
3a691f8a25 Improve our response to invalid format strings, and detect more cases.
Places that are testing for *printf failure ought to include the format
string in their error reports, since bad-format-string is one of the
more likely causes of such failure.  This both makes it easier to find
and repair the mistake, and provides at least some useful info to the
user who stumbles across such a problem.

Also, tighten snprintf.c to report EINVAL for an invalid flag or
final character in a format %-spec (including the case where the
%-spec is missing a final character altogether).  This seems like
better project policy, and it also allows removing an instruction
or two from the hot code path.

Back-patch the error reporting change in pvsnprintf, since it should be
harmless and may be helpful; but not the snprintf.c change.

Per discussion of bug #15511 from Ertuğrul Kahveci, which reported an
invalid translated format string.  These changes don't fix that error,
but they should improve matters next time we make such a mistake.

Discussion: https://postgr.es/m/15511-1d8b6a0bc874112f@postgresql.org
2018-12-06 15:08:44 -05:00
Tom Lane
6247c0bc2d Document handling of invalid/ambiguous timestamp input near DST boundaries.
The source code comments documented this, but the user-facing docs, not
so much.  Add a section to Appendix B that discusses it.

In passing, improve a couple other things in Appendix B --- notably,
a long-obsolete claim that time zone abbreviations are looked up in
a fixed table.

Per bug #15527 from Michael Davidson.

Discussion: https://postgr.es/m/15527-f1be0b4dc99ebbe7@postgresql.org
2018-11-29 18:28:11 -05:00
Tom Lane
02d3104713 Ensure static libraries have correct mod time even if ranlib messes it up.
In at least Apple's version of ranlib, the output file is updated to have
a mod time equal to the max of the timestamps of its components, and that
data only has seconds precision.  On a filesystem with sub-second file
timestamp precision --- say, APFS --- this can result in the finished
static library appearing older than its input files, which causes useless
rebuilds and possible outright failures in parallel makes.

We've only seen this reported in the field from people using Apple's
ranlib with a non-Apple make, because Apple's make doesn't know about
sub-second timestamps either so it doesn't decide rebuilds are needed.
But Apple's ranlib presumably shares code with at least some BSDen,
so it's not that unlikely that the same problem could arise elsewhere.

To fix, just "touch" the output file after ranlib finishes.

We seem to need this in only one place.  There are other calls of
ranlib in our makefiles, but they are working on intermediate files
whose timestamps are not actually important, or else on an installed
static library for which sub-second timestamp precision is unlikely
to matter either.  (Also, so far as I can tell, Apple's ranlib doesn't
mess up the file timestamp in the latter usage anyhow.)

In passing, change "ranlib" to "$(RANLIB)" in one place that was
bypassing the make macro for no good reason.

Per bug #15525 from Jack Kelly (via Alyssa Ross).
Back-patch to all supported branches.

Discussion: https://postgr.es/m/15525-a30da084f17a1faa@postgresql.org
2018-11-29 15:53:44 -05:00
Michael Paquier
b81d08d600 Fix handling of synchronous replication for stopping WAL senders
This fixes an oversight from c6c3334 which has introduced a more strict
ordering in the way WAL senders are stopped to prevent current WAL
activity when a shutdown checkpoint is created.  After all backends are
stopped, all WAL senders are requested to stop which makes them stop any
activity, and switching their state as stopping.  Once the checkpointer
knows that all WAL senders are in a stopping state, the shutdown
checkpoint can begin, with all WAL senders activated, waiting for their
clients to flush the shutdown checkpoint record.

If a subset of WAL senders are stopping and in a sync state, other WAL
senders could still be waiting for a WAL position to be synced while
committing a transaction, however the subset of stopping senders would
not release waiters, potentially breaking synchronous replication
guarantees.  This commit makes sure that even WAL senders stopping are
able to release waiters properly.

On 9.4, this can also trigger an assertion failure when setting for
example max_wal_senders to 1 where a WAL sender is not able to find
itself as in synchronous state when the instance stops.

Reported-by: Paul Guo
Author: Paul Guo, Michael Paquier
Discussion: https://postgr.es/m/CAEET0ZEv8VFqT3C-cQm6byOB4r4VYWcef1J21dOX-gcVhCSpmA@mail.gmail.com
Backpatch-through: 9.4
2018-11-29 09:13:04 +09:00
Tomas Vondra
c1a5caea82 Do not decode TOAST data for table rewrites
During table rewrites (VACUUM FULL and CLUSTER), the main heap is logged
using XLOG / FPI records, and thus (correctly) ignored in decoding.
But the associated TOAST table is WAL-logged as plain INSERT records,
and so was logically decoded and passed to reorder buffer.

That has severe consequences with TOAST tables of non-trivial size.
Firstly, reorder buffer has to keep all those changes, possibly spilling
them to a file, incurring I/O costs and disk space.

Secondly, ReoderBufferCommit() was stashing all those TOAST chunks into
a hash table, which got discarded only after processing the row from the
main heap.  But as the main heap is not decoded for rewrites, this never
happened, so all the TOAST data accumulated in memory, resulting either
in excessive memory consumption or OOM.

The fix is simple, as commit e9edc1ba already introduced infrastructure
(namely HEAP_INSERT_NO_LOGICAL flag) to skip logical decoding of TOAST
tables, but it only applied it to system tables.  So simply use it for
all TOAST data in raw_heap_insert().

That would however solve only the memory consumption issue - the TOAST
changes would still be decoded and added to the reorder buffer, and
spilled to disk (although without TOAST tuple data, so much smaller).
But we can solve that by tweaking DecodeInsert() to just ignore such
INSERT records altogether, using XLH_INSERT_CONTAINS_NEW_TUPLE flag,
instead of skipping them later in ReorderBufferCommit().

Review: Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/1a17c643-e9af-3dba-486b-fbe31bc1823a%402ndquadrant.com
Backpatch: 9.4-, where logical decoding was introduced
2018-11-28 01:53:29 +01:00
Bruce Momjian
6427bfb906 doc: fix wording for plpgsql, add "and"
Reported-by: Anthony Greene

Discussion: https://postgr.es/m/CAPRNmnsSZ4QL75FUjcS8ND_oV+WjgyPbZ4ch2RUwmW6PWzF38w@mail.gmail.com

Backpatch-through: 9.4
2018-11-26 19:41:21 -05:00
Tom Lane
74bfb5388d Fix translation of special characters in psql's LaTeX output modes.
latex_escaped_print() mistranslated \ and failed to provide any translation
for # ^ and ~, all of which would typically lead to LaTeX document syntax
errors.  In addition it didn't translate < > and |, which would typically
render as unexpected characters.

To some extent this represents shortcomings in ancient versions of LaTeX,
which if memory serves had no easy way to render these control characters
as ASCII text.  But that's been fixed for, um, decades.  In any case there
is no value in emitting guaranteed-to-fail output for these characters.

Noted while fooling with test cases added by commit 9a98984f4.  Back-patch
the code change to all supported versions.
2018-11-26 17:32:51 -05:00
Michael Paquier
0aa7e38fb7 Revert "Fix typo in documentation of toast storage"
This reverts commit 058ef3a, per complains from Magnus Hagander and Vik
Fearing.
2018-11-26 16:45:13 +09:00
Michael Paquier
efc9a3876e Fix typo in documentation of toast storage
Author: Nawaz Ahmed
Discussion: https://postgr.es/m/154319327168.1315.1846953598601966513@wrigleys.postgresql.org
2018-11-26 15:54:25 +09:00
Andrew Gierth
e5a6ae97ef Fix hstore hash function for empty hstores upgraded from 8.4.
Hstore data generated on pg 8.4 and pg_upgraded to current versions
remains in its original on-disk format unless modified. The same goes
for values generated by the addon hstore-new module on pre-9.0
versions. (The hstoreUpgrade function converts old values on the fly
when read in, but the on-disk value is not modified by this.)

Since old-format empty hstores (and hstore-new hstores) have
representations compatible with the new format, hstoreUpgrade thought
it could get away without modifying such values; but this breaks
hstore_hash (and the new hstore_hash_extended) which assumes
bit-perfect matching between semantically identical hstore values.
Only one bit actually differs (the "new version" flag in the count
field) but that of course is enough to break the hash.

Fix by making hstoreUpgrade unconditionally convert all old values to
new format.

Backpatch all the way, even though this changes a hash value in some
cases, because in those cases the hash value is already failing - for
example, a hash join between old- and new-format empty hstores will be
failing to match, or a hash index on an hstore column containing an
old-format empty value will be failing to find the value since it will
be searching for a hash derived from a new-format datum. (There are no
known field reports of this happening, probably because hashing of
hstores has only been useful in limited circumstances and there
probably isn't much upgraded data being used this way.)

Per concerns arising from discussion of commit eb6f29141be. Original
bug is my fault.

Discussion: https://postgr.es/m/60b1fd3b-7332-40f0-7e7f-f2f04f777747%402ndquadrant.com
2018-11-24 21:17:09 +00:00
Tom Lane
bf9fb00dd1 Update additional float4/8 expected-output files.
I forgot that the back branches have more variant files than HEAD :-(.
Per buildfarm.

Discussion: https://postgr.es/m/15519-4fc785b483201ff1@postgresql.org
2018-11-24 13:53:12 -05:00
Tom Lane
d5231253e3 Fix float-to-integer coercions to handle edge cases correctly.
ftoi4 and its sibling coercion functions did their overflow checks in
a way that looked superficially plausible, but actually depended on an
assumption that the MIN and MAX comparison constants can be represented
exactly in the float4 or float8 domain.  That fails in ftoi4, ftoi8,
and dtoi8, resulting in a possibility that values near the MAX limit will
be wrongly converted (to negative values) when they need to be rejected.

Also, because we compared before rounding off the fractional part,
the other three functions threw errors for values that really ought
to get rounded to the min or max integer value.

Fix by doing rint() first (requiring an assumption that it handles
NaN and Inf correctly; but dtoi8 and ftoi8 were assuming that already),
and by comparing to values that should coerce to float exactly, namely
INTxx_MIN and -INTxx_MIN.  Also remove some random cosmetic discrepancies
between these six functions.

This back-patches commits cbdb8b4c0 and 452b637d4.  In the 9.4 branch,
also back-patch the portion of 62e2a8dc2 that added PG_INTnn_MIN and
related constants to c.h, so that these functions can rely on them.

Per bug #15519 from Victor Petrovykh.

Patch by me; thanks to Andrew Gierth for analysis and discussion.

Discussion: https://postgr.es/m/15519-4fc785b483201ff1@postgresql.org
2018-11-24 12:45:50 -05:00
Andrew Gierth
2f30b311df Avoid crashes in contrib/intarray gist__int_ops (bug #15518)
1. Integer overflow in internal_size could result in memory corruption
in decompression since a zero-length array would be allocated and then
written to. This leads to crashes or corruption when traversing an
index which has been populated with sufficiently sparse values. Fix by
using int64 for computations and checking for overflow.

2. Integer overflow in g_int_compress could cause pessimal merge
choices, resulting in unnecessarily large ranges (which would in turn
trigger issue 1 above). Fix by using int64 again.

3. Even without overflow, array sizes could become large enough to
cause unexplained memory allocation errors. Fix by capping the sizes
to a safe limit and report actual errors pointing at gist__intbig_ops
as needed.

4. Large inputs to the compression function always consist of large
runs of consecutive integers, and the compression loop was processing
these one at a time in an O(N^2) manner with a lot of overhead. The
expected runtime of this function could easily exceed 6 months for a
single call as a result. Fix by performing a linear-time first pass,
which reduces the worst case to something on the order of seconds.

Backpatch all the way, since this has been wrong forever.

Per bug #15518 from report from irc user "dymk", analysis and patch by
me.

Discussion: https://postgr.es/m/15518-799e426c3b4f8358@postgresql.org
2018-11-24 08:40:02 +00:00
Bruce Momjian
e341d0336f doc: adjust time zone names text, v2
Removed one too many words.  Fix for
7906de847f229f391b9e6b5892b4b4a89f29edb4.

Reported-by: Thomas Munro

Backpatch-through: 9.4
2018-11-21 17:20:15 -05:00
Bruce Momjian
0c70dbb52e doc: adjust time zone names text
Reported-by: Kevin <kcolagio@gmail.com>

Discussion: https://postgr.es/m/154082462281.30897.14043119084654378035@wrigleys.postgresql.org

Backpatch-through: 9.4
2018-11-21 16:55:39 -05:00