programmer manual

the programmer manual describes not only how to install, setup, deploy this website project. you also find here the programming conventions of this website project and how to adapt it to the needs of your association.

to simplify the maintenance and deployment of this website project install the pjm tool on your local machine.

Hint

install the pjm tool, documented in the pjm user manual, with pip (pip install aedev-project-manager).

installation

Django CMS 3.8

the following installation steps are inspired by the YouTube video tutorial at https://www.youtube.com/watch?v=NbsRVfLCE1U

  • on a new OS/system ensure the system/apt/non-python libs needed by Django CMS are installed (see https://djangocms-installer.readthedocs.io/en/latest/libraries.html).

  • create a new python virtual environment (e.g. kairos-web39) and activate it

  • pip install djangocms-installer (version 2.0)

  • cd to your projects parent folder then run:

  • djangocms –permissions=yes –languages=en,es,de –bootstrap=yes –parent-dir=kairos kairos

  • cd kairos

  • pyenv local kairos-web39

  • open the project in PyCharm, and then:

  • set the projects python interpreter to the virtual env (kairos-web39)

  • edit requirements.txt downgrading the version of django-treebeard to 4.4 to prevent an exception on publishing (VeshRaazThapa recommends even using 4.3.1 instead of 4.4 in https://github.com/django-cms/django-cms/issues/6980)

  • open kairos/settings.py and to (1) fix the 4 inspections “Missed locally stored library for HTTP link” by downloading the external libraries and (2) reformat long lines with the hot-fix “Put attributes on separate lines”

  • open kairos/templates/base.html and replace the tag <html> with <html lang=”{{ LANGUAGE_CODE }}”>

  • open kairos/settings.py and replace in CMS_TEMPLATES feature/fullwidth/home/page with (‘base.html’, ‘Default’),

  • in kairos/templates delete feature/fullwidth/home/page.html

Reset database and migrations

Note

make a backup if you already have created page content.

  • delete migration files by running the following 2 commands in shell:

    find . -path “/migrations/.py” -not -name “__init__.py” -delete find . -path “/migrations/.pyc” -delete

  • drop your Sqlite database (with rm project.db for django cms projects)

  • create the initial migrations and generate the database schema by running these 2 commands:

    python manage.py makemigrations python manage.py migrate

  • create supervisor by running:

    python manage.py createsuperuser

  • run the server and in example.com/Administration/DJANGO CMS/User Groups(Page) create a new group ‘members’, then uncheck/remove page “add” and “update/edit” privileges defaults. finally either add to this group the permissions to add/change/delete member announcements and messages (or split/pass them to another/new group if you want to allow only some users to post messages and announcements).

  • stop the server

  • if you want to migrate members, their notifications and announcements, adapt and run the script kairos/management/commands/_members_and_announcements_data_loader.py.

  • run ./manage.py generate_thumbnails (not sure on that, because they are not displayed in filer admin).

  • run the server again and start adding the CMS pages.

Hint

for more details on database reset see by https://stackoverflow.com/questions/34576004 and https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html.

configuration and maintenance

bulk load of member and page content

as programmer you can adapt the python script _members_and_announcements_data_loader.py stored under the project path kairos/management/commands to migrate the page content and your members (including their announcements) from your old website to your new one (maintained by this project).

another option to bulk load member data from a CSV file into this website provides the Django manage.py command members_upserter. this command can also be used after the initial migration process, to add or join multiple new members.

multi-language support

this project is by default supporting the member languages english, spanish and german.

adding or removing languages, to adapt it to the needs of your association, can be done easily by changing the Django language settings. for each additional language a GNU gettext message file has to be provided (stored in this project underneath the kairos/locale folder).

multi-language messages that may be adapted for your association individually in each language, are:

  • “KairosBrandName”: brand name of your association.

  • “Time exchange * Goods exchange * Rental exchange”: sub title of the website.

  • “{member_id}-{name}”: format and content of the display name, member name or full_name of each member.

  • “{ma_cat_name}-{ma_action}-announcement”: format and content of the display name of an announcement.

don’t forget to run the Django manage.py command compilemessages after adding or changing translation texts in one of the message files.

configure member changes notification channels

this website can send notification messages automatically to an email account or to a person or a group of a messenger service (like Telegram or WhatsApp) when member has added, updated or deleted a announcement or message, or when a new member sign-up on this website.

the configuration of the used channels, services and receivers for the announcements is done by the following OS environment variables (with fallback values provided by a dotenv file named .env):

  • MCN_LOGGERS: comma-separated list of external member changes logger sinks/destinations in the format service:address=name, where name is either the name of the logger or the full_name of a member.

  • MCN_RECEIVERS: comma-separated list of messenger groups and admin members in the format service:address=name, where name is either the name of the messenger group or the full_name of a member.

  • MCN_MAIL_URI: email service, host, username and password in the format [service://][user[:password]@]mail_server_host[:mail_server_port]

  • MCN_MAIL_FROM: email sender address

  • MCN_TG_TOK: Telegram API token

  • MCN_WA_TOK: WhatsApp API token

  • MCN_WA_FROM: WhatsApp sender id

more detailed information on the content and format of these environment variables you find in the source code of kairos/utils.py and ae.notify (by searching in the code base for the environment variable name).

configure field validations

the format of data fields and the corresponding validation error messages can be configured via the settings dict FIELD_VALIDATOR_RE_MSG, where the key is the field name and the value is a tuple with the regular expression in the first item and the error message in the second item.

the following fields are configurable for your association:

  • first_name: the first name of a member in the User model. the regular expression has to prevent the comma, colon and equal characters (,:=) for the correct parsing of the notification and sign-up admin names/channels (configured in the MCN_LOGGERS, MCN_RECEIVERS and SIGNUP_ADMINS environment variables.)

  • phone: the phone number of a member, which gets stored in the User model field last_name.

configure sign-up admins

a sign-up admin is collecting the admission fees from new members.

the member-full_names and notification channels of the sign-up admins are configured by the OS environment variable SIGNUP_ADMINS, which contains a comma-separated list of all members with sign-up admin permissions in the format service:address=member_full_name.

multi-site support

the Django CMS allows to specify multiple sites.

within the sites forms, available from the Administration… menu item of the site menu, you configure the urls and display names of the site(s) of your association(s).

database queries

the following queries are useful to check, migrate or repair the content of the Django CMS database.

CMS page integrity checks can be done with the following query:

SELECT reverse_id, language, slug, path, cms_page.id as “PageId”, node_id, title, menu_title

FROM cms_title LEFT OUTER JOIN cms_page ON cms_title.page_id = cms_page.id ORDER BY reverse_id, language, slug, path;

queries to rename a Django app and model:

UPDATE django_content_type SET app_label = ‘new_app_name’ WHERE app_label = ‘old_app_name’; UPDATE django_migrations SET app = ‘new_app_name’ WHERE app = ‘old_app_name’; ALTER TABLE old_app_name_old_model_name RENAME TO new_app_name_new_model_name;

search for text in cns pages:

SELECT body FROM djangocms_text_ckeditor_text

WHERE body LIKE ‘%kairos-gomera.org%’

UNION ALL SELECT title FROM cms_title

WHERE title LIKE ‘%kairos-gomera.org%’;

delete all database migrations:

DELETE from django_migrations

add static and media folders to the code repository by running locally

this is only needed if you have added new CMS pages and new media files (pictures, PDFs, …) to your website and you want to include them into your own repository. for that execute the following shell commands:

git add -f static
git add -f media
git add -f project.db

(if you see warning on CRLF/LF conversions then extend the commands above with additional options, e.g. the first command line into git add -f -u --renormalize static)

modules and django apps

kairos

website for barter rings to communicate, exchange and share goods and time

mbr_announcements

model, view and plug-in to display, add, edit and delete member announcements

mbr_meeting_plugin

model, view and plug-in to display and edit the information of the next member meeting

mbr_messages

model, view and plug-in to display, add, edit and delete member messages

mbr_news_plugin

plug-in to display the latest changes on member announcements and message

page_link_plugin

model and plug-in to display a link to any CMS page with an associated image

deployment

the following instructions describing the deployment of this website to pythonanywhere.com.

authentication

on the pythonanywhere host navigate to account and API Token to create a new API token for the authentication. this token has then to be provided to the pjm tool either via the --web_token command line option, or in the OS environment variable AE_OPTIONS_WEB_TOKEN.

Hint

to support the specification of multiple tokens on your local machine, the name of the AE_OPTIONS_WEB_TOKEN environment variable name can be extended with hosting domain, sub-domain names and user name, in the format AE_OPTIONS_WEB_TOKEN[_AT_<sub.domain>[_<user>]]. domain and user names are converted to a valid variable name with the env_str() function (to upper-case, replacing invalid characters with underscore and prefixing upper-case characters with an underscore).

Note

the instructions in this section are using kairosgomera as the user name and kairos-web39 as the name of the virtual environment. these names can be changed to your needs.

initial deployment and configuration

this section is explaining the initial steps to prepare the new fresh pythonanywhere server for this Django project. for more details see the django tutorials of pythonanywhere, available at https://help.pythonanywhere.com/pages/FollowingTheDjangoTutorial:

  • to create a new virtual environment (e.g. kairos-web39), open a bash shell on your host and execute:

    mkvirtualenv --python=/usr/bin/python3.9 kairos-web39

  • open via the Files tab the virtualenv script at ~/.virtualenvs/kairos-web39/bin/postactivate on your host and add the following line to it (and Save it):

    set -a; source ~/.env; set +a

  • optionally correct/complete the file ~/.bashrc on your host with:

    HISTCONTROL=ignorespace:erasedups
shopt -s histappend
HISTSIZE=3000
HISTFILESIZE=6000

  • there are various ways to deploy the kairos web app server code onto your hosting server.

    if you plan to maintain, debug or change the server code, then first follow the instructions above to install this project onto your local machine. the deployment of the server code onto your host will then be done by executing the following command in a local bash shell and the projects working tree root folder:

    pjm deploy WORKTREE

    Hint

    other project versions can be deployed by specifying instead of WORKTREE a project version git tag (e.g. v1.2.3) or LATEST (for the latest stable project version).

    alternatively you could initially deploy this web app project by cloning the project repository directly onto your host. for this execute in a bash shell on your host the following commands:

    cd /home/kairosgomera git clone https://gitlab.com/ae-group/kairos

    Hint

    the first installation method will occupy much less space on your host.

    finally switch into the project root folder and install all the requirements:

    cd kairos pip install -r requirements_frozen.txt

  • open the Web tab in the interface of your pythonanywhere host and click the button manual configuration (including virtualenvs) to start a new web app.

  • in the now upcoming page under the Web tab, navigate to the Code section and specify there the paths of the Source code to /home/kairosgomera/kairos and of the Working directory to /home/kairosgomera. also specify there the used Python version (3.9).

  • in the same page (Web), in the section Code click on the link next to WSGI configuration file to adapt the WSGI file template. replace file content with the following:

    import os
import sys

from ae.base import load_dotenvs            # type: ignore


path = os.path.expanduser('~/kairos')       # results in '/home/your-username/kairos'
if path not in sys.path:
    sys.path.append(path)

load_dotenvs()

os.environ['DJANGO_SETTINGS_MODULE'] = 'kairos.settings'

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

    Hint

    more info see the section “Setting up your Web app and WSGI file” in https://help.pythonanywhere.com/pages/DeployExistingDjangoProject/.

  • specify the path of your new virtual environment in the Web tab under the section Virtualenv as /home/kairosgomera/.virtualenvs/kairos-web39.

  • add a superuser/admin to your django database by running on your host in the project root folder in a bash shell with virtual environment activated the command:

    python manage.py createsuperuser

  • upload/create a .env file to/in hosts home folder (~ or /home/user_name) containing the env vars for django (DJANGO_*) the change notification settings (MCN_*) and the data import (MEMBERS_DATA_FILE_PATH, …).

  • collect static files and put them in the folder specified by settings.STATIC_ROOT by running on your host in the project root in a bash shell with virtual env activated:

    python manage.py collectstatic

after finishing all these steps your Django site will be available at your-username.eu.pythonanywhere.com.

update from version Vx.x.xx to Vx.x.yy on pythonanywhere

  • update develop branch on github by running:

    pjm -b <branch> renew pjm prepare pjm commit pjm push pjm request pjm release LATEST pjm deploy LATEST

    Hint

    use the pjm options –web_domain

    Note

    the pjm deploy WORKTREE command can be run at any time (even before pjm prepare) e.g. to deploy unstaged/untracked changes to your test web server. only directly after the execution of the pjm release there will be no deployable changes.

    Hint

    specify --force --web_user ae-group with the request action if you don’t use a forked repository.

  • to backup the last version and db, open bash console and run from the home directory (~):

    mv kairos_Vx.x.xx/project.db backups/project.db_Vx.x.xx rm -r kairos_Vx.x.xx mv kairos kairos_Vx.x.yy git clone https://gitlab.com/ae-group/kairos cp kairos_Vx.x.yy/project.db kairos cp -r kairos_Vx.x.yy/media kairos/media cp -r kairos_Vx.x.yy/static kairos/static

    Note

    x.x.xx is the penultimate and x.x.yy is the ultimate/last web project version.

    Hint

    optionally, to save disk space on your host, change the cwd to the project root folder and then run the script ./cleanup_dev_files.sh. if you get a permission error then first make the script executable with chmod +x cleanup_dev_files.sh. when the script runs, you may need to confirm the deletion of write-protected files (mostly under the .git/ folder) by hitting the y key.

  • go to the Web tab in pythonanywhere and reload the app.

backup database, static and media files

to backup the resources of this web project, open a shell on your host, change into the root folder of this project, and run the following command:

zip -r project.db media_ini media static

configure or switch domain from siteground to pythonanywhere host

  • login into siteground

  • …. (to be continued)

DIGI zyxel router dyndns options for local/in-house deployment

  • dyndns.com

  • dtdns.com

  • no-ip.com

  • easydns.com

  • freedns.afraid.org

  • tzo.com

code conventions

general conventions for all our projects are:

  • pure python

  • fully typed (PEP 526)

  • fully documented

  • 100 % test coverage

  • multi thread save

  • code checks (using pylint and flake8)

design pattern and software principles

contributing

we want to make it as easy and fun as possible for you to contribute to this project.

reporting bugs

before you create a new issue, please check to see if you are using the latest version of this project; the bug may already be resolved.

also search for similar problems in the issue tracker; it may already be an identified problem.

include as much information as possible into the issue description, at least:

  1. version numbers (of Python and any involved packages).

  2. small self-contained code example that reproduces the bug.

  3. steps to reproduce the error.

  4. any traceback/error/log messages shown.

requesting new features

  1. on the git repository host server create new issue, providing a clear and detailed explanation of the feature you want and why it’s important to add.

  2. if you are able to implement the feature yourself (refer to the contribution steps section below).

contribution steps

thanks for your contribution – we’ll get your merge request reviewed. you could also review other merge requests, just like other developers will review yours and comment on them. based on the comments, you should address them. once the reviewers approve, the maintainers will merge.

before you start make sure you have a GitLab account.

contribution can be done either with the project-manager tool or directly by using the git command and the Gitlab server.

using the project manager tool pjm

  1. fork and clone the repository of this project to your computer

    in your console change the working directory to your project’s parent folder. then run the following command:

    pjm fork ae-group/kairos

    Note

    the pjm fork action will also add the forked repository as the upstream remote to your local repository.

    now change your current working directory to the new working-tree|project root folder, created by the pjm fork action, and execute the pjm renew action with the new_feature_or_fix part replaced by an appropriate branch name, describing shortly the new feature or the bug-fix of your contribution:

    pjm -b new_feature_or_fix renew

    this will prepare a new release version of the project and upgrade the project files created from templates to its latest version.

  2. code and check

    now use your favorite IDE/Editor to implement the new feature or code the bug fix. don’t forget to amend the project with new unit and integrity tests, and ensure they pass, by executing from time to time the pjm check action.

  3. publish your changes

    before you initiate a push/merge request against the Gitlab server, execute the pjm prepare action, which will create, with the help of the git diff command, a .commit_msg.txt file in the working tree root of your project, containing a short summary in the first line followed with a blank line and a list of the project files that got added, changed or deleted.

    Hint

    the .commit_msg.txt file can be amended by any text editor before you run the pjm commit action. for changes initiated by an issue please include the issue number (in the format fixes #<issue-number>) into this file. you may use Markdown syntax in this file for simple styling.

    to finally commit and upload your changes run the following three pjm actions in the root folder of your project:

    pjm commit
pjm push
pjm request

    the pjm commit command is first executing a pjm check action to do a finally check of the project resources and to run the unit and integrity tests. if all these checks pass then a new git commit will be created, including your changes to the project. pjm push``will then push the commit to your ``origin remote repository (your fork) and pjm request will finally create a bew merge/pull request against the upstream remote repository (the forked one).

    Hint

    to complete the workflow a maintainer of the project has to execute the pjm release action. this will merge your changes into the main branch develop of the upstream repository and then release a new version of the project onto PyPI.

more detailed information of the features of the pjm tool are available within the pjm user manual.

using git and Gitlab

alternatively to the pjm tool you could directly use the git command suite and the Gitlab website to achieve the same (with a lot more of typing and fiddling ;-):

  1. fork the upstream repository into your user account.

  2. clone your forked repo as origin remote to your computer, and add an upstream remote for the destination repo by running the following commands in the console of your local machine:

    git clone https://gitlab.com/<YourGitLabUserName>/kairos.git
git remote add upstream https://gitlab.com/ae-group/kairos.git

  3. checkout out a new local feature branch and update it to the latest version of the develop branch:

    git checkout -b <new_feature_or_fix_branch_name> develop
git pull --rebase upstream develop

    please keep your code clean by staying current with the develop branch, where code will be merged. if you find another bug, please fix it in a separated branch instead.

  4. push the branch to your fork. treat it as a backup:

    git push origin <new_feature_or_fix_branch_name>

  5. code

    implement the new feature or the bug fix; include tests, and ensure they pass.

  6. check

    run the basic code style and typing checks locally (pylint, mypy and flake8) before you commit.

  7. commit

    for every commit please write a short summary in the first line followed with a blank line and then more detailed descriptions of the change. for bug fixes please include any issue number (in the format #nnn) in your summary:

    git commit -m "issue #123: put change summary here (can be a issue title)"

    Note

    never leave the commit message blank! provide a detailed, clear, and complete description of your changes!

  8. publish your changes (prepare a Merge Request)

    before submitting a merge request, update your branch to the latest code:

    git pull --rebase upstream develop

    if you have made many commits, we ask you to squash them into atomic units of work. most issues should have one commit only, especially bug fixes, which makes them easier to back port:

    git checkout develop
git pull --rebase upstream develop
git checkout <new_feature_or_fix_branch_name>
git rebase -i develop

    push changes to your fork:

    git push -f

  9. issue/make a GitLab Merge Request:

    • navigate to your fork where you just pushed to

    • click Merge Request

    • in the branch field write your feature branch name (this is filled with your default branch name)

    • click Update Commit Range

    • ensure the changes you implemented are included in the Commits tab

    • ensure that the Files Changed tab incorporate all of your changes

    • fill in some details about your potential patch including a meaningful title

    • click New merge request.

release to PyPI

the release of a new/changed project will automatically be initiated by the GitLab CI, using the two protected vars PYPI_USERNAME and PYPI_PASSWORD (marked as masked) from the users group of this namespace, in order to provide the user name and password of the maintainers PyPI account (on Gitlab.com at Settings/CI_CD/Variables).