Improve MySQL-to-PostgreSQL migration troubleshooting docs - #9131
Improve MySQL-to-PostgreSQL migration troubleshooting docs#9131jadrales wants to merge 3 commits into
Conversation
Addresses gaps surfaced by support ticket 51867 (pgloader column-mismatch failure caused by migration-version skew): - Clarify that --run-migrations builds the Postgres schema from the specified version's migrations, not from MySQL; add guidance on confirming the actual running Mattermost version before migrating - Warn against starting the Mattermost server between schema creation and pgloader, which causes duplicate-key constraint errors - Expand the "Failed to find column or table" troubleshooting entry with root cause, real error text, and resolution steps - Add new "Duplicate key errors on a fresh target" troubleshooting section covering the non-pristine target scenario - Add new "Reloading a single failed table" section with step-by-step recovery using INCLUDING ONLY TABLE NAMES MATCHING and row-count verification Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe PostgreSQL migration guides now document Mattermost shutdown requirements, schema version alignment, pgloader result checks, version mismatch recovery, duplicate-key troubleshooting, and failed-table reload procedures. ChangesPostgreSQL migration guidance
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@source/deployment-guide/postgres-migration-assist-tool.rst`:
- Line 119: Update the Docker image recommendation in the migration guidance to
use mattermost/pgloader:latest only when the MySQL server does not require
caching_sha2_password. For servers requiring that authentication method, direct
users to the custom image or source-build path described earlier.
- Line 119: Update the migration.log guidance in the partial-load paragraph to
state that a non-zero errors value means the table load was incomplete and that
some rows may still have transferred. Retain the instruction to compare source
and target row counts and the existing recovery reference for failed tables.
- Around line 67-77: Update the final paragraph of the ``--run-migrations``
guidance to instruct users to set ``--mattermost-version`` to the actual
Mattermost version of each source environment, rather than reusing one value
across multiple environments. Preserve the existing same-line patch-release
guidance and version-confirmation instructions, while explicitly warning that
version differences can produce incorrect target schemas and pgloader failures.
In `@source/deployment-guide/postgres-migration.rst`:
- Around line 134-142: Add a warning admonition before the destructive recovery
instructions covering both recovery paths, requiring a backup or a newly created
dedicated migration database containing no required data, and requiring
Mattermost to remain stopped throughout the operation.
- Around line 134-142: Update the PostgreSQL migration procedure after the
database is dropped, recreated, and rebuilt with --run-migrations to repeat the
schema-owner setup commands before rerunning pgloader or migration-assist
postgres. Ensure the recovery sequence explicitly restores the required public
schema ownership and grants before the schema-owner pre-check.
- Around line 204-208: Update the migration instructions around the TRUNCATE
command to add a warning admonition covering its data-loss risk. Before the
command, require Mattermost to remain stopped, confirm the table is the
dedicated migration target, and check foreign-key dependencies; explicitly
caution against adding CASCADE without reviewing related tables.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4579b7f1-c737-43d3-81a8-3b42d3351642
📒 Files selected for processing (2)
source/deployment-guide/postgres-migration-assist-tool.rstsource/deployment-guide/postgres-migration.rst
| .. important:: | ||
|
|
||
| ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. | ||
|
|
||
| To confirm your actual version before migrating: | ||
|
|
||
| - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. | ||
| - **CLI**: run ``mattermost version`` on the server. | ||
|
|
||
| If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
fd -a 'postgres-migration-assist-tool\.rst' . || true
echo "== git diff stat =="
git diff --stat || true
echo "== relevant file excerpt =="
if [ -f source/deployment-guide/postgres-migration-assist-tool.rst ]; then
nl -ba source/deployment-guide/postgres-migration-assist-tool.rst | sed -n '1,140p'
fi
echo "== searches for pgloader/migration version flags =="
rg -n -- "--mattermost-version|--run-migrations|pgloader|migration" source/deployment-guide/postgres-migration-assist-tool.rst source -S || true
echo "== nearby docs mentioning multiple environments/version skew =="
rg -n -- "multiple environments|same version|pgloader|migration" source/deployment-guide/ -S || trueRepository: mattermost/docs
Length of output: 338
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="source/deployment-guide/postgres-migration-assist-tool.rst"
python3 - <<'PY'
from pathlib import Path
p = Path("source/deployment-guide/postgres-migration-assist-tool.rst")
for i, line in enumerate(p.read_text().splitlines(), 1):
if 1 <= i <= 170:
print(f"{i:4}: {line}")
PY
echo "== searches =="
rg -n -- "--mattermost-version|--run-migrations|pgloader|migration|multiple environments|same Mattermost version|version skew" source/deployment-guide/postgres-migration-assist-tool.rst source/deployment-guide -S || trueRepository: mattermost/docs
Length of output: 50371
🌐 Web query:
github mattermost migration-assist mattermost-version --mattermost-version documentation
💡 Result:
The --mattermost-version flag in the Mattermost migration-assist tool is used to specify the version of the Mattermost server for which the database migrations should be applied [1][2][3]. When using the migration-assist tool to prepare a PostgreSQL database, this flag instructs the tool to clone the specified version of the Mattermost repository, which ensures that the correct database migration files are downloaded and executed against your target database [2][3]. Usage details: 1. Flag purpose: It acts as the target version for the migration process [3]. If not provided, the tool may default to a specific version (e.g., "v8.1") [2], or require a --migrations-dir to be specified instead [2]. 2. Format: It expects a string representing the version, typically prefixed with 'v' (e.g., --mattermost-version="v10.5.4") [1][2][4]. 3. Execution example: migration-assist postgres "<POSTGRES_DSN>" --run-migrations --mattermost-version="v10.5.4" [1][5] It is recommended to use the version of Mattermost you are currently running or migrating to, ensuring compatibility with your deployment [6][3]. You can find the migration-assist tool in the official Mattermost GitHub repository [1][2][7].
Citations:
- 1: https://docs.mattermost.com/deployment-guide/postgres-migration-assist-tool.html
- 2: https://github.com/mattermost/migration-assist
- 3: http://mattermost-docs-preview-pulls.s3-website-us-east-1.amazonaws.com/7237/deploy/postgres-migration-assist-tool.html
- 4: https://forum.mattermost.com/t/solved-problem-with-migration-assist-pgloader-when-migrating-from-mysql-to-posgres/21614
- 5: Multiple issues with migration-assist and pgloader migration-assist#44
- 6: https://docs.mattermost.com/deployment-guide/postgres-migration.html
- 7: https://github.com/mattermost/migration-assist/releases
🌐 Web query:
github mattermost/migration-assist pgloader missing column same --mattermost-version multiple environments
💡 Result:
The issue of "missing columns" or "table missing" errors when using mattermost/migration-assist with pgloader is a known problem typically stemming from schema mismatches, incorrect pgloader configuration, or issues with the underlying target database schema preparation [1][2]. ### Common Causes and Solutions 1. Schema Initialization Failures: If columns or tables are missing, the target PostgreSQL database may not have been correctly initialized with the expected schema for your specific Mattermost version [2]. Ensure you have successfully run the migration-assist postgres command with the appropriate --mattermost-version flag before generating the pgloader configuration [3][4]. This command prepares the target schema [5]. 2. Search Path Issues: Incorrect quoting in DSNs (e.g., using single quotes instead of double quotes) can lead to a corrupted PostgreSQL search_path (e.g., """""""$user"""", public"""), causing the application to fail to find tables or columns [6]. Use double quotes for DSN strings when running the tool [6]. 3. Data Type Mismatches: In some cases, specific plugin tables (like calls, props, or stats) may not convert perfectly due to JSON/text type discrepancies between MySQL and PostgreSQL [7]. Manual adjustment via SQL ALTER TABLE commands (e.g., casting columns to jsonb) may be required after the initial data transfer [7]. 4. Environment Consistency: When working across multiple environments, ensure that the --mattermost-version provided to the migration-assist postgres command exactly matches the version of your source MySQL instance [8][4]. If fetching migrations fails, you can manually clone the Mattermost repository and point the tool to it using the --migrations-dir flag [8]. ### Recommended Workflow * Use the official mattermost/pgloader:latest Docker image to avoid environmental dependency issues and authentication plugin incompatibilities (e.g., caching_sha2_password in MySQL 8+) [3][2]. * Always verify the logs generated by pgloader (migration.log) to identify precisely which tables or columns are causing errors [3][2]. * If you encounter schema issues, verify the target database by running the migration-assist postgres check commands or by manually inspecting the schema in your PostgreSQL client [5][7]. If you are encountering specific errors, check if the migration-assist tool version you are using matches the documented requirements, as recent versions include automated schema checks and improvements to DSN parsing [9].
Citations:
- 1: mysql deprecation: add more troubleshooting steps #7510
- 2: https://docs.mattermost.com/deployment-guide/postgres-migration.html
- 3: https://docs.mattermost.com/deployment-guide/postgres-migration-assist-tool.html
- 4: http://mattermost-docs-preview-pulls.s3-website-us-east-1.amazonaws.com/7784/deploy/postgres-migration-assist-tool.html
- 5: https://github.com/mattermost/migration-assist
- 6: the command fails to generate correct config with single quotes for DSN migration-assist#50
- 7: Calls: post-migration issues migration-assist#18
- 8: https://forum.mattermost.com/t/database-migration-source-failure/21466
- 9: https://github.com/mattermost/migration-assist/releases
Use the source version for each environment.
--run-migrations creates the target schema for the specified Mattermost version, so each MySQL source must have its schema generated from that source’s version. If environments run different Mattermost versions, using one shared --mattermost-version value can create wrong target tables and trigger pgloader failures.
Suggested wording
- If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them.
+ If multiple environments run the same Mattermost version, use the same ``--mattermost-version`` value. Otherwise, use the version running against each environment’s source MySQL database.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| .. important:: | |
| ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. | |
| To confirm your actual version before migrating: | |
| - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. | |
| - **CLI**: run ``mattermost version`` on the server. | |
| If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. | |
| .. important:: | |
| ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. | |
| To confirm your actual version before migrating: | |
| - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. | |
| - **CLI**: run ``mattermost version`` on the server. | |
| If multiple environments run the same Mattermost version, use the same ``--mattermost-version`` value. Otherwise, use the version running against each environment’s source MySQL database. | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration-assist-tool.rst` around lines 67 -
77, Update the final paragraph of the ``--run-migrations`` guidance to instruct
users to set ``--mattermost-version`` to the actual Mattermost version of each
source environment, rather than reusing one value across multiple environments.
Preserve the existing same-line patch-release guidance and version-confirmation
instructions, while explicitly warning that version differences can produce
incorrect target schemas and pgloader failures.
| pgloader migration.load > migration.log | ||
|
|
||
| Carefully review `migration.log` for errors (e.g., duplicate-key or missing-table warnings). Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. | ||
| Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the referenced file and relevant documentation around the reported lines.
if [ -f source/deployment-guide/postgres-migration-assist-tool.rst ]; then
echo "== file exists =="
wc -l source/deployment-guide/postgres-migration-assist-tool.rst
echo "== lines 1-40 =="
sed -n '1,40p' source/deployment-guide/postgres-migration-assist-tool.rst | cat -n
echo "== lines 105-130 =="
sed -n '105,130p' source/deployment-guide/postgres-migration-assist-tool.rst | cat -n
echo "== pgloader image mentions =="
rg -n "pgloader|authentication|caching_sha2_password|Docker image|source-build|custom-image" source/deployment-guide/postgres-migration-assist-tool.rst
else
echo "file not found"
fiRepository: mattermost/docs
Length of output: 9235
Qualify the Docker image recommendation.
Line 18 says mattermost/pgloader:latest does not support MySQL’s caching_sha2_password; line 119 calls it a way to avoid auth issues. Tell users to use the Docker image only when the MySQL server does not require caching_sha2_password, and direct users who need it back to the custom image or source-build path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration-assist-tool.rst` at line 119,
Update the Docker image recommendation in the migration guidance to use
mattermost/pgloader:latest only when the MySQL server does not require
caching_sha2_password. For servers requiring that authentication method, direct
users to the custom image or source-build path described earlier.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)(postgres-migration-assist-tool\.rst|postgres-migration|pgloader|migration\.log|RELEAS|CHANGELOG|README|.*migration.*)' || true
echo "== target file around lines 90-135 =="
sed -n '90,140p' source/deployment-guide/postgres-migration-assist-tool.rst 2>/dev/null || true
echo "== search migration log summary/errors context =="
rg -n "migration\.log|summary|errors column|columns errors|did not load|rows were not transferred|row count" source docs deployment-guide 2>/dev/null || true
echo "== search pgloader migration assist references =="
rg -n "pgloader|migration assist|pg_migration|migration log|errors" . --glob '!**/.git/**' --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' 2>/dev/null | head -n 200 || true
echo "== git diff stat/name =="
git diff --stat || true
git diff --name-only || trueRepository: mattermost/docs
Length of output: 50371
🌐 Web query:
pgloader summary output errors column rows loaded behavior table load errors
💡 Result:
When running a load, pgloader displays a terminal summary table that tracks the progress and outcome of the operation [1][2]. This table includes columns for the table name, the number of rows read, the number of rows successfully imported, the number of errors encountered, and the time taken for each stage [1][2]. Error Handling and Batching Behavior pgloader uses the PostgreSQL COPY protocol for high-performance data loading [3][4][5]. Because the COPY protocol rejects an entire data set if any single part fails, pgloader manages this by splitting input data into batches (typically 25,000 rows each) [3][4][5]. When a batch is rejected: 1. pgloader parses the PostgreSQL CONTEXT error message to identify the specific row and column causing the failure [3][4][5]. 2. It then isolates the erroneous row(s) and retries the remaining rows in smaller batches to ensure maximum data recovery [3][4][5]. 3. Any rows that cannot be loaded are written to reject files [3][4][5]. Error Reporting When errors occur, pgloader generates two files in a directory named after the target database within your specified root directory [3][4][5]: -.dat file: Contains the raw data of the rejected rows [3][4][5]. -.log file: Contains the detailed PostgreSQL client-side logs related to the rejection [3][4][5]. Summary Output You can generate a machine-readable summary of the entire operation using the --summary command-line option [6][4]. By providing a filename with extensions such as.csv,.copy, or.json, you can export this summary for further automated processing [6][4]. If no extension or a standard filename is provided, the output defaults to a human-readable format [6][4].
Citations:
- 1: https://pgloader.readthedocs.io/en/latest/tutorial/mysql.html
- 2: https://pgloader.readthedocs.io/en/latest/tutorial/tutorial.html
- 3: https://pgloader.readthedocs.io/en/latest/batches.html
- 4: https://access.crunchydata.com/documentation/pgloader/latest/pdf/pgloader.pdf
- 5: https://access.crunchydata.com/documentation/pgloader/latest/pgloader/
- 6: https://pgloader.readthedocs.io/en/latest/pgloader.html
🌐 Web query:
pgloader summary table successfully imported table rows errors column means some rows transferred
💡 Result:
In the pgloader output summary table, the columns provide a breakdown of the migration process for each table or task [1][2][3]. The meaning of the columns is as follows: - table name: The name of the table, schema object, or migration phase (e.g., "fetch meta data", "Create Indexes") being processed [2][3]. - read: The number of rows (or items) read from the source database or file [2][3]. - imported: The number of rows successfully inserted into the target PostgreSQL database [2][3]. - errors: The number of rows that failed to import, typically due to data issues such as constraint violations or format mismatches [4][2][5]. - total time: The duration taken to process that specific table or task [2][3]. When the "errors" column shows a value other than zero, it indicates that some rows were rejected during the migration [4][5]. Depending on your configuration, pgloader may be set to "on error resume next" (the default for file-based loads), allowing it to continue migrating remaining rows while logging the rejected ones to separate files, typically named reject.dat and reject.log [5]. If you see "some rows transferred" alongside reported errors, it confirms that while the migration completed, it was only a partial success due to those specific rejected records [4][5]. For more detailed diagnostic information, you should check the reject files to identify the specific rows and the nature of the errors (e.g., invalid data types, null constraint violations) [4][5].
Citations:
- 1: https://manpages.ubuntu.com/manpages/trusty/man1/pgloader.1.html
- 2: https://www.scaleway.com/en/docs/tutorials/migrate-mysql-databases-postgresql-pgloader/
- 3: https://pgloader.readthedocs.io/en/latest/tutorial/mysql.html
- 4: Exit Statuses dimitri/pgloader#634
- 5: https://pgloader.readthedocs.io/
Describe partial loads accurately.
A non-zero errors value only shows that the table load was incomplete. It does not mean no rows transferred. Keep the source/target row-count check, but state that some rows may have transferred for failed tables.
Suggested wording
- any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred.
+ any table whose ``errors`` column is non-zero did not load completely; some rows may have transferred.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. | |
| Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load completely; some rows may have transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration-assist-tool.rst` at line 119,
Update the migration.log guidance in the partial-load paragraph to state that a
non-zero errors value means the table load was incomplete and that some rows may
still have transferred. Retain the instruction to compare source and target row
counts and the existing recovery reference for failed tables.
| 2. Drop and recreate the target PostgreSQL database, then rebuild the schema using the correct version: | ||
|
|
||
| .. code-block:: sh | ||
|
|
||
| migration-assist postgres "<POSTGRES_DSN>" \ | ||
| --run-migrations \ | ||
| --mattermost-version="<ACTUAL_VERSION>" | ||
|
|
||
| 3. Re-run pgloader against the freshly built schema. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add a warning before destructive recovery.
Both recovery paths can delete the target database. The instructions do not require the administrator to confirm that the database is a dedicated migration target with no required data. Add a backup or new-database requirement, and require Mattermost to remain stopped.
As per coding guidelines, use a warning admonition for meaningful data-loss risk and state the required prerequisites.
Also applies to: 175-177
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration.rst` around lines 134 - 142, Add a
warning admonition before the destructive recovery instructions covering both
recovery paths, requiring a backup or a newly created dedicated migration
database containing no required data, and requiring Mattermost to remain stopped
throughout the operation.
Source: Coding guidelines
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files:\n'
git ls-files | rg 'source/deployment-guide/(postgres-migration|postgres-migration-assist-tool)\.rst$' || true
printf '\nReference snippet postgres-migration.rst:\n'
sed -n '110,155p' source/deployment-guide/postgres-migration.rst
printf '\nRelevant snippet postgres-migration-assist-tool.rst:\n'
sed -n '35,70p' source/deployment-guide/postgres-migration-assist-tool.rst
printf '\nSearch schema owner/grants mentions:\n'
rg -n "schema-owner|owner to|OWNER TO|check-schema-owner|schema owner|pgloader|recreate|DROP DATABASE|CREATE DATABASE|run-migrations|ACTUAL_VERSION|POSTGRES_DSN|migration-assist" source/deployment-guide/postgres-migration*.rstRepository: mattermost/docs
Length of output: 21918
🌐 Web query:
PostgreSQL DROP DATABASE CREATE DATABASE default public schema owner privileges
💡 Result:
In PostgreSQL, managing databases, schemas, and ownership involves specific rules that have evolved significantly in recent versions. DATABASE OWNERSHIP AND CREATION To create a database, a user must have the CREATEDB privilege or be a superuser [1][2]. When creating a database, you can specify an owner using the OWNER clause [1][2]. If no owner is specified, it defaults to the user executing the command [1][2]. The DROP DATABASE command removes the entire database, including all its catalog entries and data files, and can only be executed by the database owner or a superuser [3]. PUBLIC SCHEMA AND PG_DATABASE_OWNER Starting with PostgreSQL 15, there was a major security change regarding the default public schema [4][5]. 1. Ownership: The public schema is now owned by a special predefined role called pg_database_owner [6][7][4]. 2. Implicit Membership: The pg_database_owner role has exactly one implicit member: the current database owner [6][7]. This ensures that the owner of a database automatically becomes the owner of that database's public schema [6][7][4]. 3. Security Upgrade: Previously, all users had CREATE and USAGE privileges on the public schema by default [4][5]. PostgreSQL 15+ revoked the default CREATE privilege from PUBLIC, meaning only the database owner can create objects in the public schema by default [4][5]. Users still retain the USAGE privilege [4]. PRIVILEGES AND ROLE MANAGEMENT When dropping a database, it is not necessary to manually drop internal schemas or tables; the DROP DATABASE command handles the cleanup of all objects contained within [3]. However, if you are attempting to drop a role (user) who owns objects within a database, you must first reassign or drop those objects [8]. - DROP OWNED: This command removes all objects within the current database owned by a specific role and revokes privileges granted to that role [9]. - REASSIGN OWNED: This command reassigns ownership of all objects owned by a role to another role [9][8]. Note that these commands must be run inside each database where the role owns objects, as they do not affect shared objects like the database itself or tablespaces [9][8]. Summary of Key Commands: - CREATE DATABASE name OWNER user_name;: Create a database with a specific owner [1][2]. - DROP DATABASE name;: Drop an entire database (requires being the owner or a superuser) [3]. - REVOKE/GRANT CREATE ON SCHEMA public FROM/TO role;: Adjust privileges for the public schema if you need to deviate from the default security model [10][11].
Citations:
- 1: https://www.postgresql.org/docs/18/sql-createdatabase.html
- 2: https://www.postgresql.org/docs/19/sql-createdatabase.html
- 3: https://www.postgresql.org/docs/18/sql-dropdatabase.html
- 4: https://www.percona.com/blog/public-schema-security-upgrade-in-postgresql-15/
- 5: https://www.depesz.com/2021/09/10/waiting-for-postgresql-15-revoke-public-create-from-public-schema-now-owned-by-pg_database_owner/
- 6: https://www.postgresql.org/docs/19/predefined-roles.html
- 7: https://www.postgresql.org/docs/18/predefined-roles.html
- 8: https://www.postgresql.org/docs/17/role-removal.html
- 9: https://www.postgresql.org/docs/18/sql-drop-owned.html
- 10: https://www.postgresql.org/docs/18/ddl-priv.html
- 11: postgres/postgres@b073c3c
🌐 Web query:
GitHub mattermost migration-assist check-schema-owner
💡 Result:
The --check-schema-owner flag is a configuration option within the Mattermost migration-assist tool, used during the migration from MySQL to PostgreSQL [1][2]. Functionality: By default, the migration-assist tool performs a pre-check to verify that the database user (e.g., mmuser) owns the public schema in the target PostgreSQL database [1]. If the migration-assist tool encounters a mismatch—where the connecting user is not the owner of the public schema—it will fail with an error such as: "could not check schema owner: the user 'mmuser' is not owner of the 'public' schema" [3][4]. Configuration: - To enable or explicitly set this check: --check-schema-owner=true [1] - To disable this check: --check-schema-owner=false [1] Resolution: If you encounter this error, it indicates that the PostgreSQL user specified in your connection string lacks the necessary ownership privileges [1][5]. You can resolve this by connecting to your PostgreSQL instance as a superuser (e.g., postgres) and executing the following commands to grant the required permissions to your database user (replace mmuser and mattermost with your actual username and database name) [3][6]: ALTER SCHEMA public OWNER TO mmuser; GRANT ALL ON SCHEMA public TO mmuser; Alternatively, you may need to grant usage and create permissions: GRANT USAGE, CREATE ON SCHEMA PUBLIC TO mmuser; Note: Some versions or workflows might also require ensuring the user is the owner of the database itself [3][6]: ALTER DATABASE mattermost OWNER TO mmuser; Always ensure your database configuration aligns with the privileges required by the migration-assist tool to successfully apply migrations [1][5].
Citations:
- 1: https://docs.mattermost.com/deployment-guide/postgres-migration-assist-tool.html
- 2: https://github.com/mattermost/migration-assist
- 3: the user "mmuser" is not owner of the "public" schema migration-assist#16
- 4: Multiple issues with migration-assist and pgloader migration-assist#44
- 5: https://support.mattermost.com/hc/en-us/articles/43338639377684-Mattermost-Upgrade-Fails-in-staging-environment-due-to-Database-Ownership-and-Migration-Errors
- 6: pq: permission denied for schema public mattermost#21442
Repeat the schema-owner setup after rebuilding the target database.
Rebuilding the PostgreSQL target with --run-migrations can leave the default public schema ownership and grants unsuitable for migration-assist postgres. For a dropped/recreated database, run the schema-owner commands again before the recovery migration-assist postgres command fails during the schema-owner pre-check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration.rst` around lines 134 - 142,
Update the PostgreSQL migration procedure after the database is dropped,
recreated, and rebuilt with --run-migrations to repeat the schema-owner setup
commands before rerunning pgloader or migration-assist postgres. Ensure the
recovery sequence explicitly restores the required public schema ownership and
grants before the schema-owner pre-check.
| 3. If the table already has partial data on the target, truncate it first: | ||
|
|
||
| .. code-block:: sql | ||
|
|
||
| TRUNCATE <table_name>; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Protect the target before TRUNCATE.
TRUNCATE <table_name> removes every row in the target table. Require Mattermost to remain stopped, confirm that the table is part of the dedicated migration target, and check foreign-key dependencies before running it. Do not add CASCADE without reviewing the related tables.
As per coding guidelines, use a warning admonition for data-loss risk and provide the prerequisites before the destructive command.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@source/deployment-guide/postgres-migration.rst` around lines 204 - 208,
Update the migration instructions around the TRUNCATE command to add a warning
admonition covering its data-loss risk. Before the command, require Mattermost
to remain stopped, confirm the table is the dedicated migration target, and
check foreign-key dependencies; explicitly caution against adding CASCADE
without reviewing related tables.
Source: Coding guidelines
esethna
added
the
preview-environment
|
Newest code from mattermost has been published to preview environment for Git SHA 4b913c1 |
|
Thanks @jadrales! The content structure and placement looks good overall, appreciate the contribution. @Combs7th can we give this a more thorough editor review after the dev reviews complete from @ewwollesen and @isacikgoz? |
2 participants
Summary
Addresses gaps surfaced by a recent support case (pgloader column-mismatch failure caused by migration-version skew):
Ticket Link
https://mattermost.zendesk.com/agent/tickets/51867