Mathesar 0.1.4

Summary

Mathesar 0.1.4 focuses on improving the installation and setup experience.

See the upgrading section below for instructions on how to upgrade to this version.

This page provides a comprehensive list of all changes in the release.

New Features

UI for configuring database connections

Now you can add, edit, and delete connections to multiple databases from within Mathesar’s UI. Previously this was only possible via editing text-based configuration.

#3170 #3223 #3299 #3309 #3319 #3326 #3341 #3348 #3349 #3352 #3354 #3356 #3368 #3377 #3387

Sample data loader

When adding a new database connection, you can choose to load sample data into that database. Sample data will be contained within specific schemas and may be useful to help new users play with Mathesar’s features.

#3368

PostgreSQL column COMMENTs

PostgreSQL COMMENT values on columns are now exposed via a read/write “description” field within Mathesar. This feature was previously available for schemas and tables and is now available for columns too.

#3186 #3219

Text-only imports

When importing CSV data, Mathesar now gives you the option to use TEXT as the database type for all columns. This choice speeds up the import for larger data sets by skipping the process of guessing column types.

#3050

We are still considering additional ways to improve performance when importing — especially for data sets with lots of columns.

Reduced database privilege installations

Mathesar can now be installed as long as the database role used during the installation has at least CONNECT and CREATE privileges on the database targeted by the installation. If you want to create a new database for Mathesar’s use, the installation will (naturally) require a role with the CREATEDB privilege.

#3117

Unified Mathesar Docker image

The published Mathesar Docker image now contains a PostgreSQL server. This is used to provide a database backend in cases where Mathesar is started via Docker without being configured to connect to any other database.

#3121 #3212

Metadata storage within SQLite

We’ve added experimental SQLite support for the storage of Mathesar metadata. This will allow brave (or foolish) users to run Mathesar with this lighter-weight DB when installing from scratch on Linux.

#3203 #3225 #2778

Improved PostgreSQL compatibility

Mathesar now officially supports, and is tested against, Postgres versions 13, 14, and 15. It’s also possible (but not yet recommended) to run Mathesar using Postgres 16.

#3206

Easier modification of sorting precedence

When you have multiple sorting conditions applied to a table, you can now rearrange them via drag and drop to adjust the precedence of the sorting conditions.

#3316

Cell values displayed within sidebar

The table sidebar features a new “Cell” tab to show the content of cells, simplifying the process of viewing large text cells.

Groundwork

• We made significant progress towards internationalizing Mathesar’s user interface. We expect to our next release to offer users the ability to toggle between English and Japanese. Subsequent releases will continue to add additional languages.

  #3102 #3103 #3104 #3302 #3321 #3337 #3340 #3350 #3389

• We began some work that will help us eventually distribute Mathesar via a Debian .deb package. Some additional work remains but we hope to introduce this installation method in a future version.

  #3189 #3225

• We implemented the backend side of a new feature to import Excel and JSON files through Mathesar’s import UI. More work still remains to implement the frontend side of this feature.

  #3083 #3195 #3132

• We took some baby steps towards building a system to automatically generate human-readable documentation for all our API endpoints. Significant work still remains.

  #3271 #3146

Documentation

• We improved and updated our documentation for installing and updating Mathesar. (#3227)

Bug fixes

• Tables having CHECK constraints are now usable within Mathesar. (#3243)
• Records can now be inserted into tables without primary keys. (#3252)
• We fixed inconsistent state when selecting a different column while editing a column’s name. (#3219)
• URL cells now retain their focus after a contained hyperlink is clicked. (#3012)
• Searching for a record via a partially-entered date string no longer gives an error. (#3343)
• The Database Page now shows loading and error indicators. (#3351)
• The Schema Page now displays more detailed information about errors encountered when loading tables and explorations. (#3323)
• Exclusion constraint violations now produce more helpful error messages. (#3200)
• Files with missing or duplicate id values can now be imported without error. (#3155)
• The record selector can now be closed by clicking on the overlay outside its modal. (#3220)
• Help text for foreign key column data types is now more accurate. (#3260)
• Users of Mathesar’s public demo site will no longer see database connections listed for other demo users. (#3129)
• More UI elements have visually distinctive focus states. (#3313)
• Date formatting is applied to arrays of date values. (#3325)
• On the record page, values within foreign key columns can now be set to NULL more intuitively. (#3310)
• A visual layout overflow bug on the record page is fixed. (#3303)
• Foreign keys referencing non-primary-key columns now display properly. (#3239)

Maintenance

• We made our CI pipeline more robust. (#3254)
• We made some updates to our workflows and developer documentation to support improvements to our issue labeling scheme. (#3338 #3298 #3280 #3336)
• We made some routine upgrades to dependencies and small adjustments to development tooling. (#3214 #3353 #3334 #3201 #3295 #3156 #3234 #3229 #3317)
• We addressed regressions from work during this release. (#3197)
• We improved error handling by preventing storing non-positive IDs for certain objects.(#3177)
• We clarified the API behavior by specifying JSON-only requests (#3090)
• We improved testing against DB objects with long names (#3140)
• We updated our org name to reflect a change from “Center of Complex Interventions” to “Mathesar Foundation”. (#3312)
• We made some improvements to our developer documentation. (#3300 #3210 #3279)
• We improved our process for generating release notes. (#3427)
• We resolved some merge conflicts after finalizing our previous release. (#3190)

Upgrading to 0.1.4

The 0.1.4 release requires more upgrade steps than we hope to have for future releases! If you run into any trouble, we encourage you to open an issue or contact us for help.

For installations using Docker Compose

If you followed our Docker Compose installation instructions, then use these steps to upgrade your installation to 0.1.4.

Note

Depending on your setup, you may need to run some commands with sudo.

1. Find needed parts

   1. Find your .env and docker-compose.yml files. Run

      docker inspect mathesar_service 
      

      and look for the value of the "com.docker.compose.project.config_files" key in the resulting JSON to find the path to the docker-compose.yml file. The .env file should be in the same directory. If you have jq installed, you can run

      docker inspect mathesar_service \
        | jq '.[0].Config.Labels."com.docker.compose.project.config_files"'
      

      and get the path directly. The .env file should be in the same directory.

   2. Copy the path of the directory containing docker-compose.yml and .env into the box below. Do not include a trailing slash.

      Then press Enter to customize this guide with the configuration directory.

   3. If you are using a Docker container for your PostgreSQL database, Run

      docker volume inspect mathesar_postgresql_data
      

      and look for the "Mountpoint" in the resulting JSON.

   4. Copy the path of the directory into the box below. Do not include a trailing slash.

      Then press Enter to customize this guide with the PostgreSQL data directory.

2. Stop Mathesar, remove old images

   docker compose -f xMATHESAR_INSTALLATION_DIRx/docker-compose.yml down --rmi all
   

3. Set up new configuration

   Warning

   MATHESAR_DATABASES has been deprecated as of v0.1.4 and will be removed entirely in future releases of Mathesar. If you end up deleting the variable from your .env file before starting up Mathesar after the upgrade, you can still add the connections manually through Mathesar’s UI.

   1. Back up the old configuration files:

      mv xMATHESAR_INSTALLATION_DIRx/docker-compose.yml xMATHESAR_INSTALLATION_DIRx/docker-compose.yml.backup
      cp xMATHESAR_INSTALLATION_DIRx/.env xMATHESAR_INSTALLATION_DIRx/env.backup
      

      (We’ll modify the old file, so we copy instead of moving it.)

   2. Download the new docker compose file:

      curl -sfL -o xMATHESAR_INSTALLATION_DIRx/docker-compose.yml https://raw.githubusercontent.com/mathesar-foundation/mathesar/0.1.4/docker-compose.yml
      

   3. Edit the xMATHESAR_INSTALLATION_DIRx/.env file to break the DJANGO_DATABASE_URL variable into its parts.

      This variable should have the form:

      DJANGO_DATABASE_URL=postgres://<username>:<password>@<host>:<port>/<database>
      

      You should edit the .env file to have the variables:

      POSTGRES_USER=<username>
      POSTGRES_PASSWORD=<password>
      POSTGRES_HOST=<host>
      POSTGRES_PORT=<port>
      POSTGRES_DB=<database>
      

      If you don’t want to set those environment variables (e.g., if they’re otherwise used), you can instead edit the docker-compose.yml file directly to add those variables.

   4. Double-check the rest of the configuration:

      • You should have your SECRET_KEY variable defined.
      • If hosting on the internet, you should have a DOMAIN_NAME variable defined.

4. Initialize new Mathesar installation

   docker compose -f xMATHESAR_INSTALLATION_DIRx/docker-compose.yml up -d
   

   This will pull new images, and start the Mathesar containers. Wait a few minutes, then run docker ps to verify that you have mathesar_service, mathesar-caddy-reverse-proxy-1, and mathesar_db running and that the service is healthy. The services should not be reporting errors. If you were not using Docker volumes for your Mathesar PostgreSQL data, you’re done, and you can login to Mathesar via your usual method. If you’re not sure, try to login to Mathesar. If you’re presented with a screen instructing you to create an Admin user, you likely need to proceed to the next step.

5. Move your PostgreSQL directory

   1. Bring down the services:

      docker compose -f xMATHESAR_INSTALLATION_DIRx/docker-compose.yml down
      

   2. Remove scaffold database data, copy your old PostgreSQL volume to the new location:

      rm -r xMATHESAR_INSTALLATION_DIRx/msar/pgdata
      cp -r xMATHESAR_PG_DIRx xMATHESAR_INSTALLATION_DIRx/msar/pgdata
      

   3. Bring the services back up:

      docker compose -f xMATHESAR_INSTALLATION_DIRx/docker-compose.yml up -d
      

6. If things look good, then you can try to login at the usual address using your normal username and password, and you should see your data.

For installations done via our guided script

If you installed Mathesar with our (now deprecated) guided script, then you have a Docker Compose installation. See the Docker Compose upgrade steps.

For installations done from scratch

If you installed Mathesar from scratch, then use these steps to upgrade your installation to 0.1.4.

Warning

These steps have not yet been tested extensively. If you run into any trouble, we encourage you to open an issue or submit a PR proposing changes to this file.

1. Go to your Mathesar installation directory.

   cd xMATHESAR_INSTALLATION_DIRx
   

   Note

   Your installation directory may be different from above if you used a different directory when installing Mathesar.

2. Pull version 0.1.4 from the repository

   git pull https://github.com/mathesar-foundation/mathesar.git
   git checkout 0.1.4
   

3. Update Python dependencies

   pip install -r requirements.txt
   

4. Next we will activate our virtual environment:

   source ./mathesar-venv/bin/activate
   

5. Update your environment variables according to the the new configuration specification. In particular, you must put the connection info for the internal DB into new POSTGRES_* variables. The DJANGO_DATABASE_URL variable is no longer supported.

6. Add the environment variables to the shell before running Django commands

   export $(sudo cat .env)
   

7. Run the latest Django migrations

   python manage.py migrate
   

8. Install the frontend dependencies

   npm ci --prefix mathesar_ui
   

9. Build the Mathesar frontend app

   npm run --prefix mathesar_ui build --max_old_space_size=4096
   

10. Update Mathesar functions on the database:

    python mathesar/install.py --skip-confirm >> /tmp/install.py.log
    

11. Restart the gunicorn server

    systemctl restart gunicorn