README.md

The COIN project
================

COIN is an Information System designed for associative ISPs in the FFDN.

![example of a coin user view : the subscriptions list](doc/_img/user-subscriptions.png)

It is written in Django, and features :
- a generic configuration interface that correspond to services provided to members
- subscriptions management
- and the corresponding billing / payment handling system

COIN can can be interfaced with the actual infrastructure to fetch the state of
services (e.g. VPS, VPN, ...) and provision them via hooks defined in the
settings.

- The project page (issue, wiki, etc) is here: https://code.ffdn.org/FFDN/coin/

- A mailing list is available, for both users and developers: https://lists.ffdn.org/wws/info/coin

- IRC : #coin-dev sur freenode / Matrix: #freenode_#coin-dev:matrix.org

License
=======

Coin is released under the GNU Affero GPL 3.0 license.  See `LICENSE` for details.

Extending Coin
==============

If you want to write your own backend, see `EXTENDING.md`.


Quickstart
==========

Virtualenv
----------

Using a virtualenv is recommended.  On Debian, install `python-virtualenv`.
On Archlinux, the package is called `python2-virtualenv`, and you must
replace the `virtualenv` command with `virtualenv2` in the following.

To create the virtualenv (the first time):

    virtualenv ./venv


To activate the virtualenv (you need to do this each time you work on
the project):

    source ./venv/bin/activate


Install dependencies. On Debian, you will probably need the
`python-dev`, `python-pip`, `libldap-dev`, `libpq-dev`, `libsasl2-dev`,
and `libjpeg-dev` packages.

    sudo apt-get install python-dev python-pip libldap2-dev libpq-dev libsasl2-dev libjpeg-dev libxml2-dev libxslt1-dev libffi-dev python-cairo libpango1.0-0

You need a recent *pip* for the installation of dependencies to work. If you
don't meet that requirement (Ubuntu trusty does not), run:

    pip install "pip>=1.5.6"

In any case, you then need to install coin python dependencies:

    pip install -r requirements.txt

You should now be able to run `python manage.py` (within the
virtualenv, obviously) without error.

Settings
--------

The `coin/settings_local.py` file is ignored by Git: feel free to override any
setting by writing into that file. For example, to override the `DEBUG`
settings:

    echo '# -*- coding: utf-8 -*-' > coin/settings_local.py
    echo 'DEBUG = TEMPLATE_DEBUG = True' >> coin/settings_local.py

See the end of this README for a reference of available configuration settings.

Development
-----------

A Dockerfile + docker-compose.yml is provided **for development purposes only**.

- Install Docker according to Docker's documentation : https://docs.docker.com/engine/install/ (or by running `apt install docker`)
- Validate that docker is running with `sudo systemctl status docker`
- You may need to add your user to the docker group to be able to run docker commands as non-root:

```
sudo usermod -aG docker $USER
# Then you need to reload your entire session (not just the terminal) for this to propagate...
# or maybe running this command does the trick: newgrp docker
```

- Install docker-compose: `sudo pip3 install docker-compose`
- Run `docker-compose up` ... the first time this will download and build a whole bunch of stuff
- You can access your app through http://localhost:8000/
- Misc tips:

```
# Launch docker-compose up, but in background
docker-compose up -d

# Check container logs after startup
docker-compose logs -f coin

# Stop / relaunch the container
docker-compose stop
docker-compose up

# Regen and apply migrations
# (ugly hack: the user coin can't write in the mounted folder ... so gotta run this as root..)
docker-compose exec -u root coin python manage.py makemigrations
docker-compose exec coin python manage.py migrate

# Enter a shell inside the container
docker-compose exec coin bash
```

- At some point you will want to initialize the admin and a bunch of dummy data :

```
docker-compose exec coin python manage.py fill_with_toy_data
docker-compose exec coin python manage.py changepassword admin
```

Running tests
-------------

There is a set of unit tests you can run with :

    DJANGO_SETTINGS_MODULE=coin.settings_test ./manage.py test

LDAP-related tests are disabled by default.

Setup LDAP parameters and activate LDAP in settings to make the LDAP tests
run.

### With pytest

Setup:

    pip install pytest-django

Run:

    pytest


Models graph
------------

*You need to have **graphviz** package installed on your system*.

You can build a huge PDF of the models used in coin with :

    ./manage.py graph_models \
      coin members offers billing resources reverse_dns configuration isp_database \
      -o models.pdf \
      --group-models

You can tweak the app list to graph (eg: if you use [optional
apps](#using-optional-apps)).

For example, if you want to graph only the `hardware_provisioning` app, run :

    ./manage.py graph_models \
      hardware_provisioning
      -o models.pdf \
      --group-models

And for more fine tunning of graph generation :

    ./manage.py graph_models --help

Available commands
==================

Some useful administration commands are available via `manage.py`.

`python manage.py members_email`: returns email addresses of all members, one
per line.  This may be useful to automatically feed a mailing list software.
Note that membership is based on the `status` field of users, not on
membership fees.  That is, even if a member has forgot to renew his or her
membership fee, his or her address will still show up in this list. More
filters are available, see the command's help for more details.

`python manage.py charge_subscriptions`: generate invoices (including a
PDF version) for each subscriber.  You probably want to run this command
every month as a cron task, see below.

`python manage.py call_for_membership_fees`: send reminder emails to members
whose membership fee is about to expire or is already expired (1 month before,
on the day of expiration, 1 month after, 2 months after, and 3 months after).
You should run this command in a cron job every day.

`python manage.py offer_subscriptions_count`: Returns subscription count grouped
by offer type.

`python manage.py import_payments_from_csv`: Import a CSV from a bank and match
payments with services and/or members. At the moment, this is quite specific to
ARN workflow

Configuration
=============

You should first setup the `sites` application, in the admin.  The domain name
configured there is used for outgoing emails.

LDAP
----

By default, LDAP support is disabled.  If you want to use LDAP (for
instance for the OpenVPN/LDAP backend, or for your own backend), see the
configuration file `coin/settings_local.example-illyse.py`.

ISP-specific configuration
--------------------------

The main place to customize Coin for your ISP is the IspDatabase application,
in the admin.  Information entered in this application has two purposes:

1) generate a JSON file usable by http://db.ffdn.org/

2) customize the user interface (postal address, bank account, ISP name & logo...)


Some bits of configuration are done in `settings.py`: LDAP branches, RSS feeds
to display on the home page, and so on.  Lastly, it is possible to override
all Coin templates (i.e. for user views and emails), see below.

Sending emails
--------------

Coin sends mails, you might want to customize the sender address:

- `DEFAULT_FROM_EMAIL` app-wide setting
- *administrative email* field, from *ISPDatabase* app, if set, will take
  precedence over `DEFAULT_FROM_EMAIL` for administrative emails (welcome email
  and membership fees), will take precedence (if filled).

Cron tasks
----------

You may want to run cron jobs for repetitive tasks.

The command that generates invoices should be run every day, so that invoices
for new subscriptions are created automatically.  On the first day of
each month, invoices will be generated for all subscriptions.
This will run the command at 23:59 every day:

    59 23 * * * /home/coin/venv/bin/python manage.py charge_subscriptions`

To send reminder emails for membership fee expiration:

    42 3 * * * /home/coin/venv/bin/python manage.py call_for_membership_fees`

If you want to be notified by email each time a reminder email is sent, you
can set an email address in the crontab:

    MAILTO=tresorier@myisp.fr

Customizing templates
---------------------

You may want to override some of the (HTML or email) templates to better suit
your structure needs.

Coin allows you to have a folder of custom templates that will contain your
templates, which gets loaded prior to coin builtins.

With this method, several templates can be overridden:

- template files in `coin/templates/`
- template files in `coin/<app_name>/templates/` for all active applications

### Do once (setup)

- Create a folder dedicated to your custom templates (*hint: outside of coin git
  tree is better*).
- Register that folder in the `EXTRA_TEMPLATE_DIRS` settings.

For instance, in `settings_local.py`:

    EXTRA_TEMPLATE_DIRS = ('/home/coin/my-folder/templates',)

### For each template you want to override

Copy the template you want to override to the right place in your custom
 folder (that's the hard part, see the example).

*Example*

Say we want to override the template located at
`coin/members/templates/members/emails/call_for_membership_fees.html` and we
set `EXTRA_TEMPLATE_DIRS = ('/home/coin/my-folder/templates',)` in settings.

Then  make a copy of the template file (and customize it) at
`/home/coin/my-folder/templates/members/emails/call_for_membership_fees.html`

Good to go :-)

Using optional apps
-------------------

Some apps are not enabled by default :

- *vpn*: Management of OpenVPN subscription and credentials through LDAP (used by [Illyse](https://www.illyse.net/) and [ARN](https://www.arn-fai.net/))
- *simple_dsl*: Simple DSL subscriptions, without handling
   any authentication backend or user configuration ("marque blanche")
- *dsl_ldap*: A DSL service which provisions LDAP for radius use (used by [Illyse](https://www.illyse.net/))
- *hardware_provisioning* : Self-service app to manage hardware inventory,
  hardware lent to members or in different stock sites.
- *maillists* : Self-service mailling-list (un)subscription for members:
  handles subscribers list for a third-party mailling-list server (sympa,
  mailman…
  etc). See [maillists app documentation (fr)](doc/admin/maillists.md).

You can enable them using the `EXTRA_INSTALLED_APPS` setting.
E.g. in `settings_local.py`:

    EXTRA_INSTALLED_APPS = (
        'vpn',
    )

If you enable an extra-app after initial installation, make sure to sync database :

    ./manage.py migrate

*nb: extra apps are loaded after the builtin apps.*


Settings
========

List of available settings in your `settings_local.py` file.

- `EXTRA_INSTALLED_APPS`: See [using optional apps](#using-optional-apps)
- `EXTRA_TEMPLATE_DIRS`: See [customizing templates](#customizing-templates)
- `LDAP_ACTIVATE`: See *LDAP*
- `MEMBER_MEMBERSHIP_INFO_URL`: Link to a page with information on how to become a member or pay the membership fee
- `MEMBERSHIP_FEE_REMINDER_DATES`: how long before/after the membership fee
  anniversary date we want to send a reminder email to the member. It defaults
  to the following:

```
MEMBERSHIP_FEE_REMINDER_DATES = [
    {'months': -3},  # 3 months before
    {'months': -2},  # 2 months before
    {'months': -1},  # 1 month before
    {'days': 0},     # the day of anniversary
    {'months': +1},  # 1 month after
]
```
- `SUBSCRIPTION_REFERENCE`: Pattern used to display a unique reference for any subscription. Helpful for bank wire transfer identification
- `REGISTRATION_OPEN` : Allow visitor to join the association by register on COIN
- `ACCOUNT_ACTIVATION_DAYS` : All account with unvalidated email will be deleted after X days
- `MEMBERSHIP_REFERENCE` : Template string to display the label the member should indicates for the bank transfer, default: "ADH-{{ user.pk }}"
- `DEFAULT_MEMBERSHIP_FEE` : Default membership fee, if you have a more complex membership fees policy, you could overwrite templates
- `PAYMENT_DELAY`: Payment delay in days for issued invoices ( default is 30 days which is the default in french law)
- `MEMBER_TERMS` : You can ask for new user to accept a terms during registration. (example:  'J\'ai lu les <a href="https://wiki.arn-fai.net/administratif:statuts" target="_BLANK">statuts de l\'association</a>' )
- `MEMBER_CAN_EDIT_PROFILE`: Allow members to edit their profiles
- `HANDLE_BALANCE`: Allows to handle money balances for members (False default)
- `INVOICES_INCLUDE_CONFIG_COMMENTS`: Add comment related to a subscription configuration when generating invoices
- `MEMBER_CAN_EDIT_VPN_CONF`: Allow members to edit some part of their vpn configuration
- `IP_ALLOCATION_MESSAGE`: Template string that will be used to log IP allocation in the corresponding coin.subnets logging system, defaults to `None` which is disabled logging, full example : `"{ip} to {member.pk} ({member.username} - {member.first_name} {member.last_name}) (for offer {offer}, {ref})"`
- `DEBUG` : Enable debug for development **do not use in production** : display
   stracktraces and enable [django-debug-toolbar](https://django-debug-toolbar.readthedocs.io).
- `SITE_TITLE`: the base of site title (displayed in browser window/tab title)
- `SITE_HEADER`: the site header (displayed in all pages as page header)
- `SITE_LOGO_URL`: URL to website logo (by default, this is an ascii-duck)
- `SUBSCRIPTIONS_NOTIFICATION_EMAILS` : Emails on which to send notifications
  uppon new registration.

### Optional apps settings

See also [using optional apps](#using-optional-apps).

#### maillists

- `MAILLIST_SYNC_COMMAND` : The command to send the list of mail addresses of a
  given mailling list to mail list server. The command will receives one
  address/line on stdin. This setting could use placholders:
    - `{email}`: the mail address of the list
    - `{short_name}`: the list name

Accounting logs
---------------

To log 'accounting-related operations' (creation/update of invoice, payment
and member balance) to a specific file, add the following to settings_local.py :

```
from settings_base import *
LOGGING["formatters"]["verbose"] = {'format': "%(asctime)s - %(name)s - %(levelname)s - %(message)s"}
LOGGING["handlers"]["coin_accounting"] = {
    'level':'INFO',
    'class':'logging.handlers.RotatingFileHandler',
    'formatter': 'verbose',
    'filename': '/var/log/coin/accounting.log',
    'maxBytes': 1024*1024*15, # 15MB
    'backupCount': 10,
}
LOGGING["loggers"]["coin.billing"]["handlers"] = [ 'coin_accounting' ]
```

External account
---------------

 - `VERBOSE_NAME_EXTERNAL_ACCOUNT` : This label is display in the member subscription view (ARN put this to 'Compte sans-nuage')
 - `VERBOSE_NAME_PLURAL_EXTERNAL_ACCOUNT` : same for plural


Deployment
==========

For real production deployment, see file `DEPLOYMENT.md`.

Testing an upgrade in a test env prior to upgrading your production
===================================================================

When upgrading from a super-old version of Coin, you may want to test the upgrade in a test env instead of yoloupgrading the production ;)

Here are some guidelines:

- Obtain a dump from your Coin's production DB
- Clone this repository, install Docker (c.f. 'Development' section)
- Make sure that you tweak the settings file to enable your relevant modules
- `mkdir ./dumps/` in the git clone, add your `.sql` dump inside, then mount it in the db's container by adding a line such as `- ./dumps/:/mnt/dumps/` in `db` > `volumes` inside `docker-compose.yml`
- Launch the `db` container with `docker-compose up -d db`, enter the container with `docker-compose exec -u root db bash`, then restore the dump with `psql -U coin coin < /mnt/dumps/yourdump.sql`
- Let's now enter the app container with `docker-compose exec coin bash`, then run migrations with `./manage.py migrate`
- Check that everything is okay (going to `http://localhost:8000/`), and/or debug if needed ;)