***************** programmer manual ***************** the programmer manual describes not only how to install, setup, deploy this website project. you also find here the programming conventions of this website project and how to adapt it to the needs of your association. to simplify the maintenance and deployment of this website project install the ``pjm`` tool on your local machine. .. hint:: install the ``pjm`` tool, documented in `the pjm user manual `__, with pip (`pip install aedev-project-manager`). installation ============ Django CMS 3.8 -------------- the following installation steps are inspired by the YouTube video tutorial at ``__ * on a new OS/system ensure the system/apt/non-python libs needed by Django CMS are installed (see ``__). * create a new python virtual environment (e.g. kairos-web39) and activate it * pip install djangocms-installer (version 2.0) * cd to your projects parent folder then run: * djangocms --permissions=yes --languages=en,es,de --bootstrap=yes --parent-dir=kairos kairos * cd kairos * pyenv local kairos-web39 * open the project in PyCharm, and then: * set the projects python interpreter to the virtual env (kairos-web39) * edit requirements.txt downgrading the version of django-treebeard to 4.4 to prevent an exception on publishing (VeshRaazThapa recommends even using 4.3.1 instead of 4.4 in ``__) * open kairos/settings.py and to (1) fix the 4 inspections “Missed locally stored library for HTTP link” by downloading the external libraries and (2) reformat long lines with the hot-fix “Put attributes on separate lines” * open kairos/templates/base.html and replace the tag with * open kairos/settings.py and replace in CMS_TEMPLATES feature/fullwidth/home/page with ('base.html', 'Default'), * in kairos/templates delete feature/fullwidth/home/page.html Reset database and migrations ----------------------------- .. note:: make a backup if you already have created page content. * delete migration files by running the following 2 commands in shell: find . -path "*/migrations/*.py" -not -name "__init__.py" -delete find . -path "*/migrations/*.pyc" -delete * drop your Sqlite database (with rm project.db for django cms projects) * create the initial migrations and generate the database schema by running these 2 commands: python manage.py makemigrations python manage.py migrate * create supervisor by running: python manage.py createsuperuser * run the server and in example.com/Administration/DJANGO CMS/User Groups(Page) create a new group 'members', then uncheck/remove page "add" and "update/edit" privileges defaults. finally either add to this group the permissions to add/change/delete member announcements and messages (or split/pass them to another/new group if you want to allow only some users to post messages and announcements). * stop the server * if you want to migrate members, their notifications and announcements, adapt and run the script kairos/management/commands/_members_and_announcements_data_loader.py. * run ./manage.py generate_thumbnails (not sure on that, because they are not displayed in filer admin). * run the server again and start adding the CMS pages. .. hint:: for more details on database reset see by ``__ and ``__. configuration and maintenance ============================= bulk load of member and page content ------------------------------------ as programmer you can adapt the python script ``_members_and_announcements_data_loader.py`` stored under the project path ``kairos/management/commands`` to migrate the page content and your members (including their announcements) from your old website to your new one (maintained by this project). another option to bulk load member data from a CSV file into this website provides the Django manage.py command ``members_upserter``. this command can also be used after the initial migration process, to add or join multiple new members. multi-language support ---------------------- this project is by default supporting the member languages ``english``, ``spanish`` and ``german``. adding or removing languages, to adapt it to the needs of your association, can be done easily by changing the `Django language settings `__. for each additional language a `GNU gettext message file `__ has to be provided (stored in this project underneath the kairos/locale folder). multi-language messages that may be adapted for your association individually in each language, are: * `"KairosBrandName"`: brand name of your association. * `"Time exchange * Goods exchange * Rental exchange"`: sub title of the website. * `"{member_id}-{name}"`: format and content of the display name, member name or full_name of each member. * `"{ma_cat_name}-{ma_action}-announcement"`: format and content of the display name of an announcement. don't forget to run the Django manage.py command ``compilemessages`` after adding or changing translation texts in one of the message files. configure member changes notification channels ---------------------------------------------- this website can send notification messages automatically to an email account or to a person or a group of a messenger service (like Telegram or WhatsApp) when member has added, updated or deleted a announcement or message, or when a new member sign-up on this website. the configuration of the used channels, services and receivers for the announcements is done by the following OS environment variables (with fallback values provided by a dotenv file named ``.env``): * `MCN_LOGGERS`: comma-separated list of external member changes logger sinks/destinations in the format service:address=name, where name is either the name of the logger or the full_name of a member. * `MCN_RECEIVERS`: comma-separated list of messenger groups and admin members in the format service:address=name, where name is either the name of the messenger group or the full_name of a member. * `MCN_MAIL_URI`: email service, host, username and password in the format [service://][user[:password]@]mail_server_host[:mail_server_port] * `MCN_MAIL_FROM`: email sender address * `MCN_TG_TOK`: Telegram API token * `MCN_WA_TOK`: WhatsApp API token * `MCN_WA_FROM`: WhatsApp sender id more detailed information on the content and format of these environment variables you find in the source code of kairos/utils.py and :mod:`ae.notify` (by searching in the code base for the environment variable name). configure field validations --------------------------- the format of data fields and the corresponding validation error messages can be configured via the settings dict `FIELD_VALIDATOR_RE_MSG`, where the key is the field name and the value is a tuple with the regular expression in the first item and the error message in the second item. the following fields are configurable for your association: * `first_name`: the first name of a member in the User model. the regular expression has to prevent the comma, colon and equal characters (,:=) for the correct parsing of the notification and sign-up admin names/channels (configured in the `MCN_LOGGERS`, `MCN_RECEIVERS` and `SIGNUP_ADMINS` environment variables.) * `phone`: the phone number of a member, which gets stored in the User model field `last_name`. configure sign-up admins ------------------------ a sign-up admin is collecting the admission fees from new members. the member-full_names and notification channels of the sign-up admins are configured by the OS environment variable `SIGNUP_ADMINS`, which contains a comma-separated list of all members with sign-up admin permissions in the format service:address=member_full_name. multi-site support ------------------ the Django CMS allows to specify multiple sites. within the sites forms, available from the `Administration...` menu item of the site menu, you configure the urls and display names of the site(s) of your association(s). database queries ---------------- the following queries are useful to check, migrate or repair the content of the Django CMS database. CMS page integrity checks can be done with the following query: SELECT reverse_id, language, slug, path, cms_page.id as "PageId", node_id, title, menu_title FROM cms_title LEFT OUTER JOIN cms_page ON cms_title.page_id = cms_page.id ORDER BY reverse_id, language, slug, path; queries to rename a Django app and model: UPDATE django_content_type SET app_label = 'new_app_name' WHERE app_label = 'old_app_name'; UPDATE django_migrations SET app = 'new_app_name' WHERE app = 'old_app_name'; ALTER TABLE old_app_name_old_model_name RENAME TO new_app_name_new_model_name; search for text in cns pages: SELECT body FROM djangocms_text_ckeditor_text WHERE body LIKE '%kairos-gomera.org%' UNION ALL SELECT title FROM cms_title WHERE title LIKE '%kairos-gomera.org%'; delete all database migrations: DELETE from django_migrations add static and media folders to the code repository by running locally ---------------------------------------------------------------------- this is only needed if you have added new CMS pages and new media files (pictures, PDFs, ...) to your website and you want to include them into your own repository. for that execute the following shell commands:: git add -f static git add -f media git add -f project.db (if you see warning on CRLF/LF conversions then extend the commands above with additional options, e.g. the first command line into :code:`git add -f -u --renormalize static`) modules and django apps ======================= .. autosummary:: :toctree: _autosummary :nosignatures: kairos mbr_announcements mbr_meeting_plugin mbr_messages mbr_news_plugin page_link_plugin deployment ========== the following instructions describing the deployment of this website to pythonanywhere.com. authentication -------------- on the pythonanywhere host navigate to ``account`` and ``API Token`` to create a new API token for the authentication. this token has then to be provided to the ``pjm`` tool either via the ``--web_token`` command line option, or in the OS environment variable AE_OPTIONS_WEB_TOKEN. .. hint:: to support the specification of multiple tokens on your local machine, the name of the AE_OPTIONS_WEB_TOKEN environment variable name can be extended with hosting domain, sub-domain names and user name, in the format ``AE_OPTIONS_WEB_TOKEN[_AT_[_]]``. domain and user names are converted to a valid variable name with the :func:`~ae.base.env_str` function (to upper-case, replacing invalid characters with underscore and prefixing upper-case characters with an underscore). .. note:: the instructions in this section are using ``kairosgomera`` as the user name and ``kairos-web39`` as the name of the virtual environment. these names can be changed to your needs. initial deployment and configuration ------------------------------------ this section is explaining the initial steps to prepare the new fresh pythonanywhere server for this Django project. for more details see the django tutorials of pythonanywhere, available at ``__: * to create a new virtual environment (e.g. kairos-web39), open a bash shell on your host and execute:: mkvirtualenv --python=/usr/bin/python3.9 kairos-web39 * open via the ``Files`` tab the virtualenv script at ~/.virtualenvs/kairos-web39/bin/postactivate on your host and add the following line to it (and Save it):: set -a; source ~/.env; set +a * optionally correct/complete the file ~/.bashrc on your host with:: HISTCONTROL=ignorespace:erasedups shopt -s histappend HISTSIZE=3000 HISTFILESIZE=6000 * there are various ways to deploy the kairos web app server code onto your hosting server. if you plan to maintain, debug or change the server code, then first :ref:`follow the instructions above to install this project onto your local machine `. the deployment of the server code onto your host will then be done by executing the following command in a local bash shell and the projects working tree root folder:: pjm deploy WORKTREE .. hint:: other project versions can be deployed by specifying instead of ``WORKTREE`` a project version git tag (e.g. ``v1.2.3``) or ``LATEST`` (for the latest stable project version). alternatively you could initially deploy this web app project by cloning the project repository directly onto your host. for this execute in a bash shell on your host the following commands: cd /home/kairosgomera git clone https://gitlab.com/ae-group/kairos .. hint:: the first installation method will occupy much less space on your host. finally switch into the project root folder and install all the requirements: cd kairos pip install -r requirements_frozen.txt * open the ``Web`` tab in the interface of your pythonanywhere host and click the button ``manual configuration (including virtualenvs)`` to start a new web app. * in the now upcoming page under the ``Web`` tab, navigate to the ``Code`` section and specify there the paths of the ``Source code`` to ``/home/kairosgomera/kairos`` and of the ``Working directory`` to ``/home/kairosgomera``. also specify there the used ``Python version`` (3.9). * in the same page (``Web``), in the section ``Code`` click on the link next to ``WSGI configuration file`` to adapt the WSGI file template. replace file content with the following:: import os import sys from ae.base import load_dotenvs # type: ignore path = os.path.expanduser('~/kairos') # results in '/home/your-username/kairos' if path not in sys.path: sys.path.append(path) load_dotenvs() os.environ['DJANGO_SETTINGS_MODULE'] = 'kairos.settings' from django.core.wsgi import get_wsgi_application application = get_wsgi_application() .. hint:: more info see the section "Setting up your Web app and WSGI file" in ``__. * specify the path of your new virtual environment in the ``Web`` tab under the section ``Virtualenv`` as ``/home/kairosgomera/.virtualenvs/kairos-web39``. * add a superuser/admin to your django database by running on your host in the project root folder in a bash shell with virtual environment activated the command:: python manage.py createsuperuser * upload/create a .env file to/in hosts home folder (~ or /home/user_name) containing the env vars for django (DJANGO_*) the change notification settings (MCN_*) and the data import (MEMBERS_DATA_FILE_PATH, ...). * collect static files and put them in the folder specified by settings.STATIC_ROOT by running on your host in the project root in a bash shell with virtual env activated:: python manage.py collectstatic after finishing all these steps your Django site will be available at your-username.eu.pythonanywhere.com. update from version Vx.x.xx to Vx.x.yy on pythonanywhere -------------------------------------------------------- * update develop branch on github by running: pjm -b renew pjm prepare pjm commit pjm push pjm request pjm release LATEST pjm deploy LATEST .. hint:: use the pjm options --web_domain .. note:: the :code:`pjm deploy WORKTREE` command can be run at any time (even before :code:`pjm prepare`) e.g. to deploy unstaged/untracked changes to your test web server. only directly after the execution of the :code:`pjm release` there will be no deployable changes. .. hint:: specify :code:`--force --web_user ae-group` with the :code:`request` action if you don't use a forked repository. * to backup the last version and db, open bash console and run from the home directory (~): mv kairos_Vx.x.xx/project.db backups/project.db_Vx.x.xx rm -r kairos_Vx.x.xx mv kairos kairos_Vx.x.yy git clone https://gitlab.com/ae-group/kairos cp kairos_Vx.x.yy/project.db kairos cp -r kairos_Vx.x.yy/media kairos/media cp -r kairos_Vx.x.yy/static kairos/static .. note:: ``x.x.xx`` is the penultimate and ``x.x.yy`` is the ultimate/last web project version. .. hint:: optionally, to save disk space on your host, change the cwd to the project root folder and then run the script ``./cleanup_dev_files.sh``. if you get a permission error then first make the script executable with ``chmod +x cleanup_dev_files.sh``. when the script runs, you may need to confirm the deletion of write-protected files (mostly under the .git/ folder) by hitting the `y` key. * go to the Web tab in pythonanywhere and reload the app. backup database, static and media files --------------------------------------- to backup the resources of this web project, open a shell on your host, change into the root folder of this project, and run the following command: zip -r project.db media_ini media static configure or switch domain from siteground to pythonanywhere host ----------------------------------------------------------------- * login into siteground * .... (to be continued) DIGI zyxel router dyndns options for local/in-house deployment -------------------------------------------------------------- - dyndns.com - dtdns.com - no-ip.com - easydns.com - freedns.afraid.org - tzo.com code conventions ================ general conventions for all our projects are: * pure python * fully typed (:pep:`526`) * fully documented * 100 % test coverage * multi thread save * code checks (using pylint and flake8) design pattern and software principles -------------------------------------- * `DRY `__ * `KISS `__ .. include:: ../CONTRIBUTING.rst Django and Django CMS documentation links ========================================= * `Django vs WordPress — Which is Better For Your Website? `__ * `Django CMS tutorial `__ * `Django CMS documentation `__ * `Mozilla Django Tutorial `__ * `Django Girls Tutorial `__ * `Getting started with Django CMS `__ * `Django's Fundament `__