Système d'information du FAI Illyse

Grégoire Jadi 6b2031dad2 hotfix: show the 'balance' field when creating a new member 1 year ago
coin 6b2031dad2 hotfix: show the 'balance' field when creating a new member 1 year ago
contrib 68975a2808 Add some Ansible scripts, thanks to Capslock from Faimaison! 3 years ago
hardware_provisioning 1cf6772200 Overload MemberAdmin to update inlines instead of monkey-patching 1 year ago
simple_dsl e2d21c21e9 Add a simple_dsl application 4 years ago
vpn f18aa84f69 Fix #14 (and a bit more) : Correct several spelling/translation errors in the admin interface 1 year ago
.gitignore c9a6a9ae8f Add a vagrant VM running a PostgreSQL database 1 year ago
DEPLOYMENT.md 0334aadd1d Use working dev default and improve the documentation for deployment in production 2 years ago
EXTENDING.md 1037a05e3a Expose the INSTALLED_APPS to templates 2 years ago
README.md f7b90ae9a6 Document import_payments_from_csv command 1 year ago
Vagrantfile c9a6a9ae8f Add a vagrant VM running a PostgreSQL database 1 year ago
manage.py 8efe3fd1fb Add some default values 5 years ago
requirements.txt 9277b03b21 Set upper bound to django-autocomplete-light's version instead of fixing it 1 year ago
setup.cfg c2e8244741 Offer pytest as testing tool 2 years ago

README.md

The COIN project

Coin is Illyse's Information System, designed to manage both members and Internet accesses, such as through DSL, VPN, wireless…

It is written in Django, and features a generic configuration interface, which allows to implement custom backends for different technologies. Currently implemented is a LDAP-based backend for OpenVPN, and a very simple DSL backend, without any authentication (useful for "white label" DSL).

Coin currently only works with python2, because python-ldap is (as of 2013) not compatible with python3.

The project page (issue, wiki, etc) is here:

https://code.ffdn.org/FFDN/coin/

A mailing list is available, for both users and developers: https://listes.illyse.org/wws/info/si

Extending Coin

If you want to write your own backend, see EXTENDING.md.

Quickstart

Virtualenv

Using a virtualenv is recommended. On Debian, install python-virtualenv. On Archlinux, the package is called python2-virtualenv, and you must replace the virtualenv command with virtualenv2 in the following.

To create the virtualenv (the first time):

virtualenv ./venv

To activate the virtualenv (you need to do this each time you work on the project):

source ./venv/bin/activate

Install dependencies. On Debian, you will probably need the python-dev, python-pip, libldap-dev, libpq-dev, libsasl2-dev, and libjpeg-dev packages.

sudo apt-get install python-dev python-pip libldap2-dev libpq-dev libsasl2-dev libjpeg-dev libxml2-dev libxslt1-dev libffi-dev

You need a recent pip for the installation of dependencies to work. If you don't meet that requirement (Ubuntu trusty does not), run:

pip install "pip>=1.5.6"

In any case, you then need to install coin python dependencies:

pip install -r requirements.txt

You should now be able to run python manage.py (within the virtualenv, obviously) without error.

Settings

The coin/settings_local.py file is ignored by Git: feel free to override any setting by writing into that file. For example, to override the DEBUG settings:

echo '# -*- coding: utf-8 -*-' > coin/settings_local.py
echo 'DEBUG = TEMPLATE_DEBUG = True' >> coin/settings_local.py

See the end of this README for a reference of available configuration settings.

Database

At this point, you should setup your database. You have two options.

With PostgreSQL (for developpement), recomended

The official database for coin is postgresql.

To ease developpement, a postgresql virtual-machine recipe is provided through vagrant.

Note: Vagrant is intended for developpement only and is totaly unsafe for a production setup.

Install requirements:

sudo apt install virtualbox vagrant

Then, to boot and configure your dev VM:

vagrant up

Default settings target that vagrant+postgreSQL setup, so, you don't have to change any setting.

With SQLite

SQLite setup may be simpler, but some features will not be available, namely:

  • automatic allocation of IP subnets (needs proper subnet implementation in the database)
  • sending automated emails to remind of expiring membership fee (needs aggregation on date fields, see Django doc)

To use sqlite instead of PostgreSQL, you have to override local settings with someting like:

DATABASES = {
    # Base de donnée du SI
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'coin.sqlite3',
        'USER': '', # Not needed for SQLite
        'PASSWORD': '', # Not needed for SQLite
        'HOST': '',  # Empty for localhost through domain sockets
        'PORT': '',  # Empty for default
    },
}

For both PostgreSQL and SQLite

The first time, you need to create the database, create a superuser, and import some base data to play with:

python manage.py migrate
python manage.py createsuperuser
python manage.py loaddata offers ip_pool # skip this if you don't use PostgreSQL

Note that the superuser will be inserted into the LDAP backend exactly in the same way as all other members, so you should use a real account (not just admin/admin).

Then, at each code update, you will only need to update dependencies and apply new migrations:

pip install -r requirements.txt
python manage.py migrate

At this point, Django should run correctly:

python manage.py runserver

Running tests

There is a set of unit tests you can run with :

DJANGO_SETTINGS_MODULE=coin.settings_test ./manage.py test

LDAP-related tests are disabled by default.

Setup LDAP parameters and activate LDAP in settings to make the LDAP tests run.

With pytest

Setup:

pip install pytest-django

Run:

pytest

Available commands

Some useful administration commands are available via manage.py.

python manage.py members_email: returns email addresses of all members, one per line. This may be useful to automatically feed a mailing list software. Note that membership is based on the status field of users, not on membership fees. That is, even if a member has forgot to renew his or her membership fee, his or her address will still show up in this list. More filters are available, see the command's help for more details.

python manage.py charge_subscriptions: generate invoices (including a PDF version) for each subscriber. You probably want to run this command every month as a cron task, see below.

python manage.py call_for_membership_fees: send reminder emails to members whose membership fee is about to expire or is already expired (1 month before, on the day of expiration, 1 month after, 2 months after, and 3 months after). You should run this command in a cron job every day.

python manage.py offer_subscriptions_count: Returns subscription count grouped by offer type.

python manage.py import_payments_from_csv: Import a CSV from a bank and match payments with services and/or members. At the moment, this is quite specific to ARN workflow

Configuration

You should first setup the sites application, in the admin. The domain name configured there is used for outgoing emails.

LDAP

By default, LDAP support is disabled. If you want to use LDAP (for instance for the OpenVPN/LDAP backend, or for your own backend), see the configuration file coin/settings_local.example-illyse.py.

ISP-specific configuration

The main place to customize Coin for your ISP is the IspDatabase application, in the admin. Information entered in this application has two purposes:

1) generate a JSON file usable by http://db.ffdn.org/

2) customize the user interface (postal address, bank account, ISP name & logo...)

Some bits of configuration are done in settings.py: LDAP branches, RSS feeds to display on the home page, and so on. Lastly, it is possible to override all Coin templates (i.e. for user views and emails), see below.

Sending emails

Coin sends mails, you might want to customize the sender address:

  • DEFAULT_FROM_EMAIL app-wide setting
  • administrative email field, from ISPDatabase app, if set, will take precedence over DEFAULT_FROM_EMAIL for administrative emails (welcome email and membership fees), will take precedence (if filled).

Cron tasks

You may want to run cron jobs for repetitive tasks.

The command that generates invoices should be run every day, so that invoices for new subscriptions are created automatically. On the first day of each month, invoices will be generated for all subscriptions. This will run the command at 23:59 every day:

59 23 * * * /home/coin/venv/bin/python manage.py charge_subscriptions`

To send reminder emails for membership fee expiration:

42 3 * * * /home/coin/venv/bin/python manage.py call_for_membership_fees`

If you want to be notified by email each time a reminder email is sent, you can set an email address in the crontab:

MAILTO=tresorier@myisp.fr

Customizing templates

You may want to override some of the (HTML or email) templates to better suit your structure needs.

Coin allows you to have a folder of custom templates that will contain your templates, which gets loaded prior to coin builtins.

With this method, several templates can be overridden:

  • template files in coin/templates/
  • template files in coin/<app_name>/templates/ for all active applications

Do once (setup)

  • Create a folder dedicated to your custom templates (hint: outside of coin git tree is better).
  • Register that folder in the EXTRA_TEMPLATE_DIRS settings.

For instance, in settings_local.py:

EXTRA_TEMPLATE_DIRS = ('/home/coin/my-folder/templates',)

For each template you want to override

Copy the template you want to override to the right place in your custom folder (that's the hard part, see the example).

Example

Say we want to override the template located at coin/members/templates/members/emails/call_for_membership_fees.html and we set EXTRA_TEMPLATE_DIRS = ('/home/coin/my-folder/templates',) in settings.

Then make a copy of the template file (and customize it) at /home/coin/my-folder/templates/members/emails/call_for_membership_fees.html

Good to go :-)

Using optional apps

Some apps are not enabled by default :

  • vpn: Management of OpenVPN subscription and credentials through LDAP
  • simple_dsl: Simple DSL subscriptions, without handling any authentication backend or user configuration ("marque blanche")
  • hardware_provisioning : Self-service app to manage hardware inventory, hardware lent to members or in different stock sites.

You can enable them using the EXTRA_INSTALLED_APPS setting. E.g. in settings_local.py:

EXTRA_INSTALLED_APPS = (
    'vpn',
)

If you enable an extra-app after initial installation, make sure to sync database :

./manage.py migrate

nb: extra apps are loaded after the builtin apps.

Settings

List of available settings in your settings_local.py file.

  • EXTRA_INSTALLED_APPS: See Customizing app list
  • EXTRA_TEMPLATE_DIRS: See Customizing templates
  • LDAP_ACTIVATE: See LDAP
  • MEMBER_MEMBERSHIP_INFO_URL: Link to a page with information on how to become a member or pay the membership fee
  • SUBSCRIPTION_REFERENCE: Pattern used to display a unique reference for any subscription. Helpful for bank wire transfer identification
  • PAYMENT_DELAY: Payment delay in days for issued invoices ( default is 30 days which is the default in french law)
  • MEMBER_CAN_EDIT_PROFILE: Allow members to edit their profiles

Accounting logs

To log 'accounting-related operations' (creation/update of invoice, payment and member balance) to a specific file, add the following to settings_local.py :

from settings_base import *
LOGGING["formatters"]["verbose"] = {'format': "%(asctime)s - %(name)s - %(levelname)s - %(message)s"}
LOGGING["handlers"]["coin_accounting"] = {
    'level':'INFO',
    'class':'logging.handlers.RotatingFileHandler',
    'formatter': 'verbose',
    'filename': '/var/log/coin/accounting.log',
    'maxBytes': 1024*1024*15, # 15MB
    'backupCount': 10,
}
LOGGING["loggers"]["coin.billing"]["handlers"] = [ 'coin_accounting' ]

More information

For the rest of the setup (database, LDAP), see https://doc.illyse.net/projects/ils-si/wiki/Mise_en_place_environnement_de_dev

For real production deployment, see file DEPLOYMENT.md.