Merge pull request #2 from mailcow/master

update local mailcow

.github/ISSUE_TEMPLATE/Bug_report.md

@@ -3,28 +3,63 @@ name: Bug report
 about: Report a bug for this project
 
 ---
+<!--
+  For community support and other discussions, you are welcome to visit us on our community channels listed at https://mailcow.github.io/mailcow-dockerized-docs/#community-support. For professional commercial support, please check out https://mailcow.github.io/mailcow-dockerized-docs/#commercial-support instead
+-->
 
-**README and remove me**
+**Prior to placing the issue, please check following:** *(fill out each checkbox with a `X` once done)*
-For community support and other discussion, you are welcome to visit and stay with us @ Freenode, #mailcow
+- [ ] I understand that not following below instructions might result in immediate closing and deletion of my issue.
-Answering can take a few seconds up to many hours, please be patient.
+- [ ] I have understood that answers are voluntary and community-driven, and not commercial support.
-Commercial support, including a ticket system, can be found @ https://www.servercow.de/mailcow#support - we are also available via Telegram. \o/
+- [ ] I have verified that my issue has not been already answered in the past. I also checked previous [issues](https://github.com/mailcow/mailcow-dockerized/issues).
 
-**Describe the bug, try to make it reproducible**
+---
-A clear and concise description of what the bug is. How can it be reproduced? 
-If applicable, add screenshots to help explain your problem. Very useful for bugs in mailcow UI.
 
-**System information and quick debugging**
+**Description of the bug**: What kind of issue have you *exactly* come across?
-General logs:
+<!--
-- Please take a look at the [documentation](https://mailcow.github.io/mailcow-dockerized-docs/debug-logs/).
+  This should be a clear and concise description of what the bug is. What EXACTLY does happen?
+  If applicable, add screenshots to help explain your problem. Very useful for bugs in mailcow UI.
+  Write your detailed description below.
+-->
+
+My issue is...
+
+**Reproduction of said bug**: How *exactly* do you reproduce the bug?
+<!--
+  Here it is really helpful to know how exactly you are able to reproduce the reported issue.
+  Meaning: What are the exact steps - one by one - to get the above described behavior.
+  Screenshots can be added, if helpful. Add the text below.
+-->
+
+1. I go to...
+2. And then to...
+3. But once I do...
+
+__I have tried or I do...__ *(fill out each checkbox with a `X` if applicable)*
+- [ ] In case of WebUI issue, I have tried clearing the browser cache and the issue persists.
+- [ ] I do run mailcow on a Synology, QNAP or any other sort of NAS.
+
+**System information**
+<!--
+  In this stage we would kindly ask you to attach logs or general system information about your setup.
+  Please carefully read the questions and instructions below.
+-->
 
 Further information (where applicable):
- - Your OS (is Apparmor or SELinux active?)
+
- - Your virtualization technology (KVM/QEMU, Xen, VMware, VirtualBox etc.)
+| Question | Answer |
- - Your server/VM specifications (Memory, CPU Cores)
+| --- | --- |
- - Don't try to run mailcow on a Synology or QNAP NAS, do you?
+| My operating system | I_DO_REPLY_HERE |
- - Docker and Docker Compose versions
+| Is Apparmor, SELinux or similar active? | I_DO_REPLY_HERE |
- - Output of `git diff origin/master`, any other changes to the code?
+| Virtualization technlogy (KVM, VMware, Xen, etc) | I_DO_REPLY_HERE |
+| Server/VM specifications (Memory, CPU Cores) | I_DO_REPLY_HERE |
+| Docker Version (`docker version`) | I_DO_REPLY_HERE |
+| Docker-Compose Version (`docker-compose version`) | I_DO_REPLY_HERE |
+| Reverse proxy (custom solution) | I_DO_REPLY_HERE |
+
+Further notes:
+ - Output of `git diff origin/master`, any other changes to the code? If so, please post them.
  - All third-party firewalls and custom iptables rules are unsupported. Please check the Docker docs about how to use Docker with your own ruleset. Nevertheless, iptabels output can help _us_ to help _you_: `iptables -L -vn`, `ip6tables -L -vn`, `iptables -L -vn -t nat` and `ip6tables -L -vn -t nat `
- - Reverse proxy? If you think this problem is related to your reverse proxy, please post your configuration.
- - Browser (if it's a Web UI issue) - please clean your browser cache and try again, problem persists?
  - Check `docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254` (set the IP accordingly, if you changed the internal mailcow network) and `docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @1.1.1.1` - output? Timeout?
+ 
+ General logs:
+- Please take a look at the [official documentation](https://mailcow.github.io/mailcow-dockerized-docs/debug-logs/).

.gitignore

@@ -1,10 +1,12 @@
 rebuild-images.sh
 data/conf/sogo/sieve.creds
+data/conf/phpfpm/sogo-sso/sogo-sso.pass
 data/conf/dovecot/dovecot-master.passwd
 data/conf/dovecot/dovecot-master.userdb
 mailcow.conf
 mailcow.conf_backup
 data/conf/nginx/*.active
+data/conf/postfix/extra.cf
 data/conf/postfix/sql
 data/conf/postfix/allow_mailcow_local.regexp
 data/conf/dovecot/sql
@@ -24,11 +26,15 @@ data/conf/nginx/*.custom
 data/conf/nginx/*.bak
 data/conf/dovecot/acl_anyone
 data/conf/dovecot/mail_plugins*
+data/conf/dovecot/sogo-sso.conf
 data/conf/dovecot/extra.conf
+data/conf/dovecot/shared_namespace.conf
 data/conf/rspamd/custom/*
 data/conf/portainer/
 data/gitea/
 data/gogs/
 data/conf/sogo/plist_ldap
+update_diffs/
 .github/
 docker-compose.override.yml
+refresh_images.sh

README.md

@@ -2,7 +2,7 @@
 
 ## Want to support mailcow?
 
-Donate via **PayPal** [![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=JWBSYHF4SMC68) or via **Liberapay** [![Liberapay.com](https://mailcow.email/img/lp.png)](https://liberapay.com/mailcow)
+Please [consider a support contract (around 30 € per month) with Servercow](https://www.servercow.de/mailcow#support) to support further development. _We_ support _you_ while _you_ support _us_. :)
 
 Or just spread the word: moo.
 

data/Dockerfiles/acme/Dockerfile

@@ -1,8 +1,9 @@
-FROM alpine:3.9
+FROM alpine:3.10
 
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
-RUN apk add --update --no-cache \
+RUN apk upgrade --no-cache \
+  && apk add --update --no-cache \
   bash \
   curl \
   openssl \
@@ -12,9 +13,9 @@ RUN apk add --update --no-cache \
   redis \
   tini \
   tzdata \
-  py-pip \
+  python3 \
-  && pip install --upgrade pip \
+  && python3 -m pip install --upgrade pip \
-  && pip install acme-tiny
+  && python3 -m pip install acme-tiny
 
 COPY docker-entrypoint.sh /srv/docker-entrypoint.sh
 COPY expand6.sh /srv/expand6.sh

data/Dockerfiles/acme/docker-entrypoint.sh

@@ -5,6 +5,21 @@ exec 5>&1
 # Thanks to https://github.com/cvmiller -> https://github.com/cvmiller/expand6
 source /srv/expand6.sh
 
+# Skipping IP check when we like to live dangerously
+if [[ "${SKIP_IP_CHECK}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+  SKIP_IP_CHECK=y
+fi
+
+# Skipping HTTP check when we like to live dangerously
+if [[ "${SKIP_HTTP_VERIFICATION}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+  SKIP_HTTP_VERIFICATION=y
+fi
+
+# Request certificate for MAILCOW_HOSTNAME ony
+if [[ "${ONLY_MAILCOW_HOSTNAME}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+  ONLY_MAILCOW_HOSTNAME=y
+fi
+
 log_f() {
   if [[ ${2} == "no_nl" ]]; then
     echo -n "$(date) - ${1}"
@@ -42,7 +57,6 @@ mkdir -p ${ACME_BASE}/acme
 [[ -f ${ACME_BASE}/acme/private/privkey.pem ]] && mv ${ACME_BASE}/acme/private/privkey.pem ${ACME_BASE}/acme/key.pem
 [[ -f ${ACME_BASE}/acme/private/account.key ]] && mv ${ACME_BASE}/acme/private/account.key ${ACME_BASE}/acme/account.pem
 
-
 reload_configurations(){
   # Reading container IDs
   # Wrapping as array to ensure trimmed content when calling $NGINX etc.
@@ -118,21 +132,25 @@ get_ipv6(){
 }
 
 verify_challenge_path(){
+  if [[ ${SKIP_HTTP_VERIFICATION} == "y" ]]; then
+    echo '(skipping check, returning 0)'
+    return 0
+  fi
   # verify_challenge_path URL 4|6
-  RAND_FILE=${RANDOM}${RANDOM}${RANDOM}
+  RANDOM_N=${RANDOM}${RANDOM}${RANDOM}
-  touch /var/www/acme/${RAND_FILE}
+  echo ${RANDOM_N} > /var/www/acme/${RANDOM_N}
-  if [[ "$(curl -${2} http://${1}/.well-known/acme-challenge/${RAND_FILE} --write-out %{http_code} --silent --output /dev/null)" =~ ^(2|3)  ]]; then
+  if [[ "$(curl --insecure -${2} -L http://${1}/.well-known/acme-challenge/${RANDOM_N} --silent)" == "${RANDOM_N}"  ]]; then
-    rm /var/www/acme/${RAND_FILE}
+    rm /var/www/acme/${RANDOM_N}
     return 0
   else
-    rm /var/www/acme/${RAND_FILE}
+    rm /var/www/acme/${RANDOM_N}
     return 1
   fi
 }
 
 [[ ! -f ${ACME_BASE}/dhparams.pem ]] && cp ${SSL_EXAMPLE}/dhparams.pem ${ACME_BASE}/dhparams.pem
 
-if [[ -f ${ACME_BASE}/cert.pem ]] && [[ -f ${ACME_BASE}/key.pem ]]; then
+if [[ -f ${ACME_BASE}/cert.pem ]] && [[ -f ${ACME_BASE}/key.pem ]] && [[ $(stat -c%s ${ACME_BASE}/cert.pem) != 0 ]]; then
   ISSUER=$(openssl x509 -in ${ACME_BASE}/cert.pem -noout -issuer)
   if [[ ${ISSUER} != *"Let's Encrypt"* && ${ISSUER} != *"mailcow"* && ${ISSUER} != *"Fake LE Intermediate"* ]]; then
     log_f "Found certificate with issuer other than mailcow snake-oil CA and Let's Encrypt, skipping ACME client..."
@@ -156,6 +174,7 @@ else
     exec env TRIGGER_RESTART=1 $(readlink -f "$0")
   fi
 fi
+chmod 600 ${ACME_BASE}/key.pem
 
 log_f "Waiting for database... " no_nl
 while ! mysqladmin status --socket=/var/run/mysqld/mysqld.sock -u${DBUSER} -p${DBPASS} --silent; do
@@ -196,10 +215,8 @@ while true; do
     log_f "Using existing Lets Encrypt account key ${ACME_BASE}/acme/account.pem"
   fi
 
-  # Skipping IP check when we like to live dangerously
+  chmod 600 ${ACME_BASE}/acme/key.pem
-  if [[ "${SKIP_IP_CHECK}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+  chmod 600 ${ACME_BASE}/acme/account.pem
-    SKIP_IP_CHECK=y
-  fi
 
   # Cleaning up and init validation arrays
   unset SQL_DOMAIN_ARR
@@ -228,7 +245,7 @@ while true; do
       ADDITIONAL_SAN_ARR+=($i)
     fi
   done
-  ADDITIONAL_WC_ARR+=('autodiscover')
+  ADDITIONAL_WC_ARR+=('autodiscover' 'autoconfig')
 
   # Start IP detection
   log_f "Detecting IP addresses... " no_nl
@@ -255,9 +272,10 @@ while true; do
     SQL_DOMAIN_ARR+=("${domains}")
   done < <(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e "SELECT domain FROM domain WHERE backupmx=0" -Bs)
 
+  if [[ ${ONLY_MAILCOW_HOSTNAME} != "y" ]]; then
   for SQL_DOMAIN in "${SQL_DOMAIN_ARR[@]}"; do
     for SUBDOMAIN in "${ADDITIONAL_WC_ARR[@]}"; do
-      if [[  "${SUBDOMAIN}.${SQL_DOMAIN}" != "${MAILCOW_HOSTNAME}" ]]; then
+      if [[ "${SUBDOMAIN}.${SQL_DOMAIN}" != "${MAILCOW_HOSTNAME}" ]]; then
         A_SUBDOMAIN=$(dig A ${SUBDOMAIN}.${SQL_DOMAIN} +short | tail -n 1)
         AAAA_SUBDOMAIN=$(dig AAAA ${SUBDOMAIN}.${SQL_DOMAIN} +short | tail -n 1)
         # Check if CNAME without v6 enabled target
@@ -268,10 +286,10 @@ while true; do
           log_f "Found AAAA record for ${SUBDOMAIN}.${SQL_DOMAIN}: ${AAAA_SUBDOMAIN} - skipping A record check"
           if [[ $(expand ${IPV6:-"0000:0000:0000:0000:0000:0000:0000:0000"}) == $(expand ${AAAA_SUBDOMAIN}) ]] || [[ ${SKIP_IP_CHECK} == "y" ]]; then
             if verify_challenge_path "${SUBDOMAIN}.${SQL_DOMAIN}" 6; then
-              log_f "Confirmed AAAA record ${AAAA_SUBDOMAIN}"
+              log_f "Confirmed AAAA record with IP ${AAAA_SUBDOMAIN}, adding SAN"
               VALIDATED_CONFIG_DOMAINS+=("${SUBDOMAIN}.${SQL_DOMAIN}")
             else
-              log_f "Confirmed AAAA record ${AAAA_SUBDOMAIN}, but HTTP validation failed"
+              log_f "Confirmed AAAA record with IP ${AAAA_SUBDOMAIN}, but HTTP validation failed"
             fi
           else
             log_f "Cannot match your IP ${IPV6:-NO_IPV6_LINK} against hostname ${SUBDOMAIN}.${SQL_DOMAIN} ($(expand ${AAAA_SUBDOMAIN}))"
@@ -280,10 +298,10 @@ while true; do
           log_f "Found A record for ${SUBDOMAIN}.${SQL_DOMAIN}: ${A_SUBDOMAIN}"
           if [[ ${IPV4:-ERR} == ${A_SUBDOMAIN} ]] || [[ ${SKIP_IP_CHECK} == "y" ]]; then
             if verify_challenge_path "${SUBDOMAIN}.${SQL_DOMAIN}" 4; then
-              log_f "Confirmed A record ${A_SUBDOMAIN}"
+              log_f "Confirmed A record ${A_SUBDOMAIN}, adding SAN"
               VALIDATED_CONFIG_DOMAINS+=("${SUBDOMAIN}.${SQL_DOMAIN}")
             else
-              log_f "Confirmed AAAA record ${A_SUBDOMAIN}, but HTTP validation failed"
+              log_f "Confirmed A record with IP ${A_SUBDOMAIN}, but HTTP validation failed"
             fi
           else
             log_f "Cannot match your IP ${IPV4} against hostname ${SUBDOMAIN}.${SQL_DOMAIN} (${A_SUBDOMAIN})"
@@ -294,6 +312,7 @@ while true; do
       fi
     done
   done
+  fi
 
   A_MAILCOW_HOSTNAME=$(dig A ${MAILCOW_HOSTNAME} +short | tail -n 1)
   AAAA_MAILCOW_HOSTNAME=$(dig AAAA ${MAILCOW_HOSTNAME} +short | tail -n 1)
@@ -308,10 +327,10 @@ while true; do
         log_f "Confirmed AAAA record ${AAAA_MAILCOW_HOSTNAME}"
         VALIDATED_MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
       else
-        log_f "Confirmed AAAA record ${A_MAILCOW_HOSTNAME}, but HTTP validation failed"
+        log_f "Confirmed AAAA record with IP ${AAAA_MAILCOW_HOSTNAME}, but HTTP validation failed"
       fi
     else
-      log_f "Cannot match your IP ${IPV6:-NO_IPV6_LINK} against hostname ${MAILCOW_HOSTNAME} ($(expand ${AAAA_MAILCOW_HOSTNAME}))"
+      log_f "Cannot match your IP ${IPV6:-NO_IPV6_LINK} against hostname ${MAILCOW_HOSTNAME} (DNS returned $(expand ${AAAA_MAILCOW_HOSTNAME}))"
     fi
   elif [[ ! -z ${A_MAILCOW_HOSTNAME} ]]; then
     log_f "Found A record for ${MAILCOW_HOSTNAME}: ${A_MAILCOW_HOSTNAME}"
@@ -320,15 +339,16 @@ while true; do
         log_f "Confirmed A record ${A_MAILCOW_HOSTNAME}"
         VALIDATED_MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
       else
-        log_f "Confirmed A record ${A_MAILCOW_HOSTNAME}, but HTTP validation failed"
+        log_f "Confirmed A record with IP ${A_MAILCOW_HOSTNAME}, but HTTP validation failed"
       fi
     else
-      log_f "Cannot match your IP ${IPV4} against hostname ${MAILCOW_HOSTNAME} (${A_MAILCOW_HOSTNAME})"
+      log_f "Cannot match your IP ${IPV4} against hostname ${MAILCOW_HOSTNAME} (DNS returned ${A_MAILCOW_HOSTNAME})"
     fi
   else
     log_f "No A or AAAA record found for hostname ${MAILCOW_HOSTNAME}"
   fi
 
+  if [[ ${ONLY_MAILCOW_HOSTNAME} != "y" ]]; then
   for SAN in "${ADDITIONAL_SAN_ARR[@]}"; do
     # Skip on CAA errors for SAN
     SAN_PARENT_DOMAIN=$(echo ${SAN} | cut -d. -f2-)
@@ -354,13 +374,13 @@ while true; do
       log_f "Found AAAA record for ${SAN}: ${AAAA_SAN} - skipping A record check"
       if [[ $(expand ${IPV6:-"0000:0000:0000:0000:0000:0000:0000:0000"}) == $(expand ${AAAA_SAN}) ]] || [[ ${SKIP_IP_CHECK} == "y" ]]; then
         if verify_challenge_path "${SAN}" 6; then
-          log_f "Confirmed AAAA record ${AAAA_SAN}"
+          log_f "Confirmed AAAA record with IP ${AAAA_SAN}"
           ADDITIONAL_VALIDATED_SAN+=("${SAN}")
         else
-          log_f "Confirmed AAAA record ${AAAA_SAN}, but HTTP validation failed"
+          log_f "Confirmed AAAA record with IP ${AAAA_SAN}, but HTTP validation failed"
         fi
       else
-        log_f "Cannot match your IP ${IPV6:-NO_IPV6_LINK} against hostname ${SAN} ($(expand ${AAAA_SAN}))"
+        log_f "Cannot match your IP ${IPV6:-NO_IPV6_LINK} against hostname ${SAN} (DNS returned $(expand ${AAAA_SAN}))"
       fi
     elif [[ ! -z ${A_SAN} ]]; then
       log_f "Found A record for ${SAN}: ${A_SAN}"
@@ -369,21 +389,23 @@ while true; do
           log_f "Confirmed A record ${A_SAN}"
           ADDITIONAL_VALIDATED_SAN+=("${SAN}")
         else
-          log_f "Confirmed A record ${A_SAN}, but HTTP validation failed"
+          log_f "Confirmed A record with IP ${A_SAN}, but HTTP validation failed"
         fi
       else
-        log_f "Cannot match your IP ${IPV4} against hostname ${SAN} (${A_SAN})"
+        log_f "Cannot match your IP ${IPV4} against hostname ${SAN} (DNS returned ${A_SAN})"
       fi
     else
       log_f "No A or AAAA record found for hostname ${SAN}"
     fi
   done
+  fi
 
   # Unique elements
   ALL_VALIDATED=(${VALIDATED_MAILCOW_HOSTNAME} $(echo ${VALIDATED_CONFIG_DOMAINS[*]} ${ADDITIONAL_VALIDATED_SAN[*]} | xargs -n1 | sort -u | xargs))
   if [[ -z ${ALL_VALIDATED[*]} ]]; then
     log_f "Cannot validate hostnames, skipping Let's Encrypt for 1 hour."
     log_f "Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently."
+    redis-cli -h redis SET ACME_FAIL_TIME "$(date +%s)"
     sleep 1h
     exec $(readlink -f "$0")
   fi
@@ -397,19 +419,19 @@ while true; do
   # Finding difference in SAN array now vs. SAN array by current configuration
   array_diff ORPHANED_SAN SAN_ARRAY_NOW ALL_VALIDATED
   if [[ ! -z ${ORPHANED_SAN[*]} ]]; then
-    log_f "Found orphaned SANs ${ORPHANED_SAN[*]}"
+    log_f "Found orphaned SAN ${ORPHANED_SAN[*]}"
     SAN_CHANGE=1
   fi
   array_diff ADDED_SAN ALL_VALIDATED SAN_ARRAY_NOW
   if [[ ! -z ${ADDED_SAN[*]} ]]; then
-    log_f "Found new SANs ${ADDED_SAN[*]}"
+    log_f "Found new SAN ${ADDED_SAN[*]}"
     SAN_CHANGE=1
   fi
 
   if [[ ${SAN_CHANGE} == 0 ]]; then
     # Certificate did not change but could be due for renewal (4 weeks)
-    if ! openssl x509 -checkend 1209600 -noout -in ${ACME_BASE}/cert.pem; then
+    if ! openssl x509 -checkend 2592000 -noout -in ${ACME_BASE}/cert.pem; then
-      log_f "Certificate is due for renewal (< 2 weeks)"
+      log_f "Certificate is due for renewal (< 30 days)"
     else
       log_f "Certificate validation done, neither changed nor due for renewal, sleeping for another day."
       sleep 1d
@@ -462,7 +484,7 @@ while true; do
         cp ${ACME_BASE}/acme/cert.pem ${ACME_BASE}/cert.pem
         cp ${ACME_BASE}/acme/key.pem ${ACME_BASE}/key.pem
         reload_configurations
-        rm /var/www/acme/*
+        rm /var/www/acme/* 2> /dev/null
         log_f "Certificate successfully deployed, removing backup, sleeping 1d"
         sleep 1d
       else
@@ -476,6 +498,7 @@ while true; do
       ACME_RESPONSE_B64=$(echo "${ACME_RESPONSE}" | openssl enc -e -A -base64)
       log_f "${ACME_RESPONSE_B64}" redis_only b64
       log_f "Retrying in 30 minutes..."
+      redis-cli -h redis SET ACME_FAIL_TIME "$(date +%s)"
       sleep 30m
       exec $(readlink -f "$0")
       ;;

data/Dockerfiles/clamd/Dockerfile

@@ -3,7 +3,7 @@ FROM debian:stretch-slim
 LABEL maintainer "André Peters <andre.peters@servercow.de>"
 
 # Installation
-ENV CLAMAV 0.101.1
+ENV CLAMAV 0.101.4
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
   ca-certificates \

data/Dockerfiles/clamd/bootstrap.sh

@@ -48,6 +48,7 @@ while true; do
   sleep 2m
   SANE_MIRRORS="$(dig +ignore +short rsync.sanesecurity.net)"
   for sane_mirror in ${SANE_MIRRORS}; do
+    CE=
     rsync -avp --chown=clamav:clamav --chmod=Du=rwx,Dgo=rx,Fu=rw,Fog=r --timeout=5 rsync://${sane_mirror}/sanesecurity/ \
       --include 'blurl.ndb' \
       --include 'junk.ndb' \
@@ -61,7 +62,9 @@ while true; do
       --include 'sanesecurity.ftm' \
       --include 'sigwhitelist.ign2' \
       --exclude='*' /var/lib/clamav/
-    if [ $? -eq 0 ]; then
+    CE=$?
+    chmod 755 /var/lib/clamav/
+    if [ ${CE} -eq 0 ]; then
       echo RELOAD | nc localhost 3310
       break
     fi

data/Dockerfiles/dockerapi/Dockerfile

@@ -1,11 +1,12 @@
-FROM alpine:3.9
+FROM alpine:3.10
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
-RUN apk add -U --no-cache python2 python-dev py-pip gcc musl-dev tzdata openssl-dev libffi-dev \
+WORKDIR /app
-  && pip2 install --upgrade pip \
-  && pip2 install --upgrade docker==3.0.1 flask flask-restful pyOpenSSL \
-  && apk del python-dev py2-pip gcc
 
-COPY server.py /
+RUN apk add --update --no-cache python3 openssl tzdata \
+ && pip3 install --upgrade pip \
+ && pip3 install --upgrade docker flask flask-restful
 
-CMD ["python2", "-u", "/server.py"]
+COPY server.py /app/
+
+CMD ["python3", "-u", "/app/server.py"]

data/Dockerfiles/dockerapi/server.py

@@ -1,10 +1,11 @@
+#!/usr/bin/env python3
+
 from flask import Flask
 from flask_restful import Resource, Api
 from flask import jsonify
 from flask import Response
 from flask import request
 from threading import Thread
-from OpenSSL import crypto
 import docker
 import uuid
 import signal
@@ -14,6 +15,8 @@ import re
 import sys
 import ssl
 import socket
+import subprocess
+import traceback
 
 docker_client = docker.DockerClient(base_url='unix://var/run/docker.sock', version='auto')
 app = Flask(__name__)
@@ -43,262 +46,317 @@ class container_get(Resource):
 class container_post(Resource):
   def post(self, container_id, post_action):
     if container_id and container_id.isalnum() and post_action:
-      if post_action == 'stop':
+      try:
-        try:
+        """Dispatch container_post api call"""
-          for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+        if post_action == 'exec':
-            container.stop()
+          if not request.json or not 'cmd' in request.json:
-          return jsonify(type='success', msg='command completed successfully')
+            return jsonify(type='danger', msg='cmd is missing')
-        except Exception as e:
+          if not request.json or not 'task' in request.json:
-          return jsonify(type='danger', msg=str(e))
+            return jsonify(type='danger', msg='task is missing')
-
-      elif post_action == 'start':
-        try:
-          for container in docker_client.containers.list(all=True, filters={"id": container_id}):
-            container.start()
-          return jsonify(type='success', msg='command completed successfully')
-        except Exception as e:
-          return jsonify(type='danger', msg=str(e))
-
-      elif post_action == 'restart':
-        try:
-          for container in docker_client.containers.list(all=True, filters={"id": container_id}):
-            container.restart()
-          return jsonify(type='success', msg='command completed successfully')
-        except Exception as e:
-          return jsonify(type='danger', msg=str(e))
-
-      elif post_action == 'top':
-        try:
-          for container in docker_client.containers.list(all=True, filters={"id": container_id}):
-            return jsonify(type='success', msg=container.top())
-        except Exception as e:
-          return jsonify(type='danger', msg=str(e))
-
-      elif post_action == 'stats':
-        try:
-          for container in docker_client.containers.list(all=True, filters={"id": container_id}):
-            return jsonify(type='success', msg=container.stats(decode=True, stream=False))
-        except Exception as e:
-          return jsonify(type='danger', msg=str(e))
-
-      elif post_action == 'exec':
-
-        if not request.json or not 'cmd' in request.json:
-          return jsonify(type='danger', msg='cmd is missing')
-
-        if request.json['cmd'] == 'mailq':
-          if 'items' in request.json:
-            r = re.compile("^[0-9a-fA-F]+$")
-            filtered_qids = filter(r.match, request.json['items'])
-            if filtered_qids:
-              if request.json['task'] == 'delete':
-                flagged_qids = ['-d %s' % i for i in filtered_qids]
-                sanitized_string = str(' '.join(flagged_qids));
-                try:
-                  for container in docker_client.containers.list(filters={"id": container_id}):
-                    postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
-                    return exec_run_handler('generic', postsuper_r)
-                except Exception as e:
-                  return jsonify(type='danger', msg=str(e))
-              if request.json['task'] == 'hold':
-                flagged_qids = ['-h %s' % i for i in filtered_qids]
-                sanitized_string = str(' '.join(flagged_qids));
-                try:
-                  for container in docker_client.containers.list(filters={"id": container_id}):
-                    postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
-                    return exec_run_handler('generic', postsuper_r)
-                except Exception as e:
-                  return jsonify(type='danger', msg=str(e))
-              if request.json['task'] == 'unhold':
-                flagged_qids = ['-H %s' % i for i in filtered_qids]
-                sanitized_string = str(' '.join(flagged_qids));
-                try:
-                  for container in docker_client.containers.list(filters={"id": container_id}):
-                    postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
-                    return exec_run_handler('generic', postsuper_r)
-                except Exception as e:
-                  return jsonify(type='danger', msg=str(e))
-              if request.json['task'] == 'deliver':
-                flagged_qids = ['-i %s' % i for i in filtered_qids]
-                try:
-                  for container in docker_client.containers.list(filters={"id": container_id}):
-                    for i in flagged_qids:
-                      postqueue_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postqueue " + i], user='postfix')
-                      # todo: check each exit code
-                    return jsonify(type='success', msg=str("Scheduled immediate delivery"))
-                except Exception as e:
-                  return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'list':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                mailq_return = container.exec_run(["/usr/sbin/postqueue", "-j"], user='postfix')
-                return exec_run_handler('utf8_text_only', mailq_return)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'flush':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                postqueue_r = container.exec_run(["/usr/sbin/postqueue", "-f"], user='postfix')
-                return exec_run_handler('generic', postqueue_r)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'super_delete':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                postsuper_r = container.exec_run(["/usr/sbin/postsuper", "-d", "ALL"])
-                return exec_run_handler('generic', postsuper_r)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-
-        elif request.json['cmd'] == 'system':
-          if request.json['task'] == 'fts_rescan':
-            if 'username' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  rescan_return = container.exec_run(["/bin/bash", "-c", "/usr/local/bin/doveadm fts rescan -u '" + request.json['username'].replace("'", "'\\''") + "'"], user='vmail')
-                  if rescan_return.exit_code == 0:
-                    return jsonify(type='success', msg='fts_rescan: rescan triggered')
-                  else:
-                    return jsonify(type='warning', msg='fts_rescan error')
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-            if 'all' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  rescan_return = container.exec_run(["/bin/bash", "-c", "/usr/local/bin/doveadm fts rescan -A"], user='vmail')
-                  if rescan_return.exit_code == 0:
-                    return jsonify(type='success', msg='fts_rescan: rescan triggered')
-                  else:
-                    return jsonify(type='warning', msg='fts_rescan error')
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'df':
-            if 'dir' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  df_return = container.exec_run(["/bin/bash", "-c", "/bin/df -H '" + request.json['dir'].replace("'", "'\\''") + "' | /usr/bin/tail -n1 | /usr/bin/tr -s [:blank:] | /usr/bin/tr ' ' ','"], user='nobody')
-                  if df_return.exit_code == 0:
-                    return df_return.output.rstrip()
-                  else:
-                    return "0,0,0,0,0,0"
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'mysql_upgrade':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                sql_shell = container.exec_run(["/bin/bash"], stdin=True, socket=True, user='mysql')
-                upgrade_cmd = "/usr/bin/mysql_upgrade -uroot -p'" + os.environ['DBROOT'].replace("'", "'\\''") + "'\n"
-                sql_socket = sql_shell.output;
-                try :
-                  sql_socket.sendall(upgrade_cmd.encode('utf-8'))
-                  sql_socket.shutdown(socket.SHUT_WR)
-                except socket.error:
-                  return jsonify(type='danger', msg=str('socket error'))
-                worker_response = recv_socket_data(sql_socket)
-                matched = False
-                for line in worker_response.split("\n"):
-                  if 'is already upgraded to' in line:
-                    matched = True
-                if matched:
-                  return jsonify(type='success', msg='mysql_upgrade: already upgraded')
-                else:
-                  container.restart()
-                  return jsonify(type='warning', msg='mysql_upgrade: upgrade was applied')
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-
-        elif request.json['cmd'] == 'reload':
-          if request.json['task'] == 'dovecot':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                reload_return = container.exec_run(["/bin/bash", "-c", "/usr/local/sbin/dovecot reload"])
-                return exec_run_handler('generic', reload_return)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-          if request.json['task'] == 'postfix':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                reload_return = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postfix reload"])
-                return exec_run_handler('generic', reload_return)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-          if request.json['task'] == 'nginx':
-            try:
-              for container in docker_client.containers.list(filters={"id": container_id}):
-                reload_return = container.exec_run(["/bin/sh", "-c", "/usr/sbin/nginx -s reload"])
-                return exec_run_handler('generic', reload_return)
-            except Exception as e:
-              return jsonify(type='danger', msg=str(e))
-
-        elif request.json['cmd'] == 'sieve':
-          if request.json['task'] == 'list':
-            if 'username' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  sieve_return = container.exec_run(["/bin/bash", "-c", "/usr/local/bin/doveadm sieve list -u '" + request.json['username'].replace("'", "'\\''") + "'"])
-                  return exec_run_handler('utf8_text_only', sieve_return)
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-          elif request.json['task'] == 'print':
-            if 'username' in request.json and 'script_name' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  sieve_return = container.exec_run(["/bin/bash", "-c", "/usr/local/bin/doveadm sieve get -u '" + request.json['username'].replace("'", "'\\''") + "' '" + request.json['script_name'].replace("'", "'\\''") + "'"])
-                  return exec_run_handler('utf8_text_only', sieve_return)
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-
-        elif request.json['cmd'] == 'maildir':
-          if request.json['task'] == 'cleanup':
-            if 'maildir' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  sane_name = re.sub(r'\W+', '', request.json['maildir'])
-                  maildir_cleanup = container.exec_run(["/bin/bash", "-c", "if [[ -d '/var/vmail/" + request.json['maildir'].replace("'", "'\\''") + "' ]]; then /bin/mv '/var/vmail/" + request.json['maildir'].replace("'", "'\\''") + "' '/var/vmail/_garbage/" + str(int(time.time())) + "_" + sane_name + "'; fi"], user='vmail')
-                  return exec_run_handler('generic', maildir_cleanup)
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
-
-        elif request.json['cmd'] == 'rspamd':
-          if request.json['task'] == 'worker_password':
-            if 'raw' in request.json:
-              try:
-                for container in docker_client.containers.list(filters={"id": container_id}):
-                  worker_shell = container.exec_run(["/bin/bash"], stdin=True, socket=True, user='_rspamd')
-                  worker_cmd = "/usr/bin/rspamadm pw -e -p '" + request.json['raw'].replace("'", "'\\''") + "' 2> /dev/null\n"
-                  worker_socket = worker_shell.output;
-                  try :
-                    worker_socket.sendall(worker_cmd.encode('utf-8'))
-                    worker_socket.shutdown(socket.SHUT_WR)
-                  except socket.error:
-                    return jsonify(type='danger', msg=str('socket error'))
-                  worker_response = recv_socket_data(worker_socket)
-                  matched = False
-                  for line in worker_response.split("\n"):
-                    if '$2$' in line:
-                      matched = True
-                      hash = line.strip()
-                      hash_out = re.search('\$2\$.+$', hash).group(0)
-                      f = open("/access.inc", "w")
-                      f.write('enable_password = "' + re.sub('[^0-9a-zA-Z\$]+', '', hash_out.rstrip()) + '";\n')
-                      f.close()
-                      container.restart()
-                  if matched:
-                    return jsonify(type='success', msg='command completed successfully')
-                  else:
-                    return jsonify(type='danger', msg='command did not complete')
-              except Exception as e:
-                return jsonify(type='danger', msg=str(e))
 
+          api_call_method_name = '__'.join(['container_post', str(post_action), str(request.json['cmd']), str(request.json['task']) ])
         else:
-          return jsonify(type='danger', msg='Unknown command')
+          api_call_method_name = '__'.join(['container_post', str(post_action) ])
 
-      else:
+        api_call_method = getattr(self, api_call_method_name, lambda container_id: jsonify(type='danger', msg='container_post - unknown api call'))
-        return jsonify(type='danger', msg='invalid action')
+
+
+        print("api call: %s, container_id: %s" % (api_call_method_name, container_id))
+        return api_call_method(container_id)
+      except Exception as e:
+        print("error - container_post: %s" % str(e))  
+        return jsonify(type='danger', msg=str(e))
 
     else:
       return jsonify(type='danger', msg='invalid container id or missing action')        
 
+
+  # api call: container_post - post_action: stop
+  def container_post__stop(self, container_id):
+    for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+      container.stop()
+    return jsonify(type='success', msg='command completed successfully')
+
+
+  # api call: container_post - post_action: start
+  def container_post__start(self, container_id):
+    for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+      container.start()
+    return jsonify(type='success', msg='command completed successfully')
+
+
+  # api call: container_post - post_action: restart
+  def container_post__restart(self, container_id):
+    for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+      container.restart()
+    return jsonify(type='success', msg='command completed successfully')
+
+
+  # api call: container_post - post_action: top
+  def container_post__top(self, container_id):
+    for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+      return jsonify(type='success', msg=container.top())
+
+
+  # api call: container_post - post_action: stats
+  def container_post__stats(self, container_id):
+    for container in docker_client.containers.list(all=True, filters={"id": container_id}):
+      for stat in container.stats(decode=True, stream=True):
+        return jsonify(type='success', msg=stat )
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: delete
+  def container_post__exec__mailq__delete(self, container_id):
+    if 'items' in request.json:  
+      r = re.compile("^[0-9a-fA-F]+$")
+      filtered_qids = filter(r.match, request.json['items'])
+      if filtered_qids:
+        flagged_qids = ['-d %s' % i for i in filtered_qids]
+        sanitized_string = str(' '.join(flagged_qids));
+
+        for container in docker_client.containers.list(filters={"id": container_id}):
+          postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
+          return exec_run_handler('generic', postsuper_r)
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: hold
+  def container_post__exec__mailq__hold(self, container_id):
+    if 'items' in request.json:  
+      r = re.compile("^[0-9a-fA-F]+$")
+      filtered_qids = filter(r.match, request.json['items'])
+      if filtered_qids:
+        flagged_qids = ['-h %s' % i for i in filtered_qids]
+        sanitized_string = str(' '.join(flagged_qids));
+
+        for container in docker_client.containers.list(filters={"id": container_id}):
+          postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
+          return exec_run_handler('generic', postsuper_r)
+
+
+   # api call: container_post - post_action: exec - cmd: mailq - task: unhold
+  def container_post__exec__mailq__unhold(self, container_id):
+    if 'items' in request.json:  
+      r = re.compile("^[0-9a-fA-F]+$")
+      filtered_qids = filter(r.match, request.json['items'])
+      if filtered_qids:
+        flagged_qids = ['-H %s' % i for i in filtered_qids]
+        sanitized_string = str(' '.join(flagged_qids));
+
+        for container in docker_client.containers.list(filters={"id": container_id}):
+          postsuper_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postsuper " + sanitized_string])
+          return exec_run_handler('generic', postsuper_r)
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: deliver
+  def container_post__exec__mailq__deliver(self, container_id):
+    if 'items' in request.json:  
+      r = re.compile("^[0-9a-fA-F]+$")
+      filtered_qids = filter(r.match, request.json['items'])
+      if filtered_qids:
+        flagged_qids = ['-i %s' % i for i in filtered_qids]
+
+        for container in docker_client.containers.list(filters={"id": container_id}):
+          for i in flagged_qids:
+            postqueue_r = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postqueue " + i], user='postfix')
+            # todo: check each exit code
+          return jsonify(type='success', msg=str("Scheduled immediate delivery"))
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: list
+  def container_post__exec__mailq__list(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      mailq_return = container.exec_run(["/usr/sbin/postqueue", "-j"], user='postfix')
+      return exec_run_handler('utf8_text_only', mailq_return)
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: flush
+  def container_post__exec__mailq__flush(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      postqueue_r = container.exec_run(["/usr/sbin/postqueue", "-f"], user='postfix')
+      return exec_run_handler('generic', postqueue_r)
+
+
+  # api call: container_post - post_action: exec - cmd: mailq - task: super_delete
+  def container_post__exec__mailq__super_delete(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      postsuper_r = container.exec_run(["/usr/sbin/postsuper", "-d", "ALL"])
+      return exec_run_handler('generic', postsuper_r)
+
+
+  # api call: container_post - post_action: exec - cmd: system - task: fts_rescan
+  def container_post__exec__system__fts_rescan(self, container_id):
+    if 'username' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        rescan_return = container.exec_run(["/bin/bash", "-c", "/usr/bin/doveadm fts rescan -u '" + request.json['username'].replace("'", "'\\''") + "'"], user='vmail')
+        if rescan_return.exit_code == 0:
+          return jsonify(type='success', msg='fts_rescan: rescan triggered')
+        else:
+          return jsonify(type='warning', msg='fts_rescan error')
+
+    if 'all' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        rescan_return = container.exec_run(["/bin/bash", "-c", "/usr/bin/doveadm fts rescan -A"], user='vmail')
+        if rescan_return.exit_code == 0:
+          return jsonify(type='success', msg='fts_rescan: rescan triggered')
+        else:
+          return jsonify(type='warning', msg='fts_rescan error')
+
+
+  # api call: container_post - post_action: exec - cmd: system - task: df
+  def container_post__exec__system__df(self, container_id):
+    if 'dir' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        df_return = container.exec_run(["/bin/bash", "-c", "/bin/df -H '" + request.json['dir'].replace("'", "'\\''") + "' | /usr/bin/tail -n1 | /usr/bin/tr -s [:blank:] | /usr/bin/tr ' ' ','"], user='nobody')
+        if df_return.exit_code == 0:
+          return df_return.output.decode('utf-8').rstrip()
+        else:
+          return "0,0,0,0,0,0"
+
+
+  # api call: container_post - post_action: exec - cmd: system - task: mysql_upgrade
+  def container_post__exec__system__mysql_upgrade(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      cmd = "/usr/bin/mysql_upgrade -uroot -p'" + os.environ['DBROOT'].replace("'", "'\\''") + "'\n"
+      cmd_response = exec_cmd_container(container, cmd, user='mysql')
+
+      matched = False
+      for line in cmd_response.split("\n"):
+        if 'is already upgraded to' in line:
+          matched = True
+      if matched:
+        return jsonify(type='success', msg='mysql_upgrade: already upgraded')
+      else:
+        container.restart()
+        return jsonify(type='warning', msg='mysql_upgrade: upgrade was applied')
+
+
+  # api call: container_post - post_action: exec - cmd: reload - task: dovecot
+  def container_post__exec__reload__dovecot(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      reload_return = container.exec_run(["/bin/bash", "-c", "/usr/sbin/dovecot reload"])
+      return exec_run_handler('generic', reload_return)
+
+
+  # api call: container_post - post_action: exec - cmd: reload - task: postfix
+  def container_post__exec__reload__postfix(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      reload_return = container.exec_run(["/bin/bash", "-c", "/usr/sbin/postfix reload"])
+      return exec_run_handler('generic', reload_return)
+
+
+  # api call: container_post - post_action: exec - cmd: reload - task: nginx
+  def container_post__exec__reload__nginx(self, container_id):
+    for container in docker_client.containers.list(filters={"id": container_id}):
+      reload_return = container.exec_run(["/bin/sh", "-c", "/usr/sbin/nginx -s reload"])
+      return exec_run_handler('generic', reload_return)
+
+
+  # api call: container_post - post_action: exec - cmd: sieve - task: list
+  def container_post__exec__sieve__list(self, container_id):
+    if 'username' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        sieve_return = container.exec_run(["/bin/bash", "-c", "/usr/bin/doveadm sieve list -u '" + request.json['username'].replace("'", "'\\''") + "'"])
+        return exec_run_handler('utf8_text_only', sieve_return)
+
+
+  # api call: container_post - post_action: exec - cmd: sieve - task: print
+  def container_post__exec__sieve__print(self, container_id):
+    if 'username' in request.json and 'script_name' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        cmd = ["/bin/bash", "-c", "/usr/bin/doveadm sieve get -u '" + request.json['username'].replace("'", "'\\''") + "' '" + request.json['script_name'].replace("'", "'\\''") + "'"]  
+        sieve_return = container.exec_run(cmd)
+        return exec_run_handler('utf8_text_only', sieve_return)
+
+
+  # api call: container_post - post_action: exec - cmd: maildir - task: cleanup
+  def container_post__exec__maildir__cleanup(self, container_id):
+    if 'maildir' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        sane_name = re.sub(r'\W+', '', request.json['maildir'])
+        cmd = ["/bin/bash", "-c", "if [[ -d '/var/vmail/" + request.json['maildir'].replace("'", "'\\''") + "' ]]; then /bin/mv '/var/vmail/" + request.json['maildir'].replace("'", "'\\''") + "' '/var/vmail/_garbage/" + str(int(time.time())) + "_" + sane_name + "'; fi"]
+        maildir_cleanup = container.exec_run(cmd, user='vmail')
+        return exec_run_handler('generic', maildir_cleanup)
+
+
+
+  # api call: container_post - post_action: exec - cmd: rspamd - task: worker_password
+  def container_post__exec__rspamd__worker_password(self, container_id):
+    if 'raw' in request.json:
+      for container in docker_client.containers.list(filters={"id": container_id}):
+        cmd = "/usr/bin/rspamadm pw -e -p '" + request.json['raw'].replace("'", "'\\''") + "' 2> /dev/null"
+        cmd_response = exec_cmd_container(container, cmd, user="_rspamd")
+        
+        matched = False
+        for line in cmd_response.split("\n"):
+          if '$2$' in line:
+            hash = line.strip()
+            hash_out = re.search('\$2\$.+$', hash).group(0)
+            rspamd_passphrase_hash = re.sub('[^0-9a-zA-Z\$]+', '', hash_out.rstrip())
+
+            rspamd_password_filename = "/etc/rspamd/override.d/worker-controller-password.inc"
+            cmd = '''/bin/echo 'enable_password = "%s";' > %s && cat %s''' % (rspamd_passphrase_hash, rspamd_password_filename, rspamd_password_filename)
+            cmd_response = exec_cmd_container(container, cmd, user="_rspamd")
+
+            if rspamd_passphrase_hash.startswith("$2$") and rspamd_passphrase_hash in cmd_response:
+              container.restart()
+              matched = True
+
+        if matched:
+            return jsonify(type='success', msg='command completed successfully')
+        else:
+            return jsonify(type='danger', msg='command did not complete')
+        
+
+def exec_cmd_container(container, cmd, user, timeout=2, shell_cmd="/bin/bash"):
+
+  def recv_socket_data(c_socket, timeout):
+    c_socket.setblocking(0)
+    total_data=[];
+    data='';
+    begin=time.time()
+    while True:
+      if total_data and time.time()-begin > timeout:
+        break
+      elif time.time()-begin > timeout*2:
+        break
+      try:
+        data = c_socket.recv(8192)
+        if data:
+          total_data.append(data.decode('utf-8'))
+          #change the beginning time for measurement
+          begin=time.time()
+        else:
+          #sleep for sometime to indicate a gap
+          time.sleep(0.1)
+          break
+      except:
+        pass
+    return ''.join(total_data)
+    
+  try :
+    socket = container.exec_run([shell_cmd], stdin=True, socket=True, user=user).output._sock
+    if not cmd.endswith("\n"):
+      cmd = cmd + "\n"
+    socket.send(cmd.encode('utf-8'))
+    data = recv_socket_data(socket, timeout)
+    socket.close()
+    return data
+
+  except Exception as e:
+    print("error - exec_cmd_container: %s" % str(e))
+    traceback.print_exc(file=sys.stdout)
+
+def exec_run_handler(type, output):
+  if type == 'generic':
+    if output.exit_code == 0:
+      return jsonify(type='success', msg='command completed successfully')
+    else:
+      return jsonify(type='danger', msg='command failed: ' + output.output.decode('utf-8'))
+  if type == 'utf8_text_only':
+    r = Response(response=output.output.decode('utf-8'), status=200, mimetype="text/plain")
+    r.headers["Content-Type"] = "text/plain; charset=utf-8"
+    return r
+
 class GracefulKiller:
   kill_now = False
   def __init__(self):
@@ -308,84 +366,26 @@ class GracefulKiller:
   def exit_gracefully(self, signum, frame):
     self.kill_now = True
 
+def create_self_signed_cert():
+    process = subprocess.Popen(
+      "openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout /app/dockerapi_key.pem -out /app/dockerapi_cert.pem -subj /CN=dockerapi/O=mailcow -addext subjectAltName=DNS:dockerapi".split(),
+      stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=False
+    )
+    process.wait()
+
 def startFlaskAPI():
   create_self_signed_cert()
   try:
     ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
     ctx.check_hostname = False
-    ctx.load_cert_chain(certfile='/cert.pem', keyfile='/key.pem')
+    ctx.load_cert_chain(certfile='/app/dockerapi_cert.pem', keyfile='/app/dockerapi_key.pem')
   except:
-    print "Cannot initialize TLS, retrying in 5s..."
+    print ("Cannot initialize TLS, retrying in 5s...")
     time.sleep(5)
   app.run(debug=False, host='0.0.0.0', port=443, threaded=True, ssl_context=ctx)
 
-def recv_socket_data(c_socket, timeout=10):
-  c_socket.setblocking(0)
-  total_data=[];
-  data='';
-  begin=time.time()
-  while True:
-    if total_data and time.time()-begin > timeout:
-      break
-    elif time.time()-begin > timeout*2:
-      break
-    try:
-      data = c_socket.recv(8192)
-      if data:
-        total_data.append(data)
-        #change the beginning time for measurement
-        begin=time.time()
-      else:
-        #sleep for sometime to indicate a gap
-        time.sleep(0.1)
-        break
-    except:
-      pass
-  return ''.join(total_data)
-
-def exec_run_handler(type, output):
-  if type == 'generic':
-    if output.exit_code == 0:
-      return jsonify(type='success', msg='command completed successfully')
-    else:
-      return jsonify(type='danger', msg='command failed: ' + output.output)
-  if type == 'utf8_text_only':
-    r = Response(response=output.output, status=200, mimetype="text/plain")
-    r.headers["Content-Type"] = "text/plain; charset=utf-8"
-    return r
-
-def create_self_signed_cert():
-  success = False
-  while not success:
-    try:
-      pkey = crypto.PKey()
-      pkey.generate_key(crypto.TYPE_RSA, 2048)
-      cert = crypto.X509()
-      cert.get_subject().O = "mailcow"
-      cert.get_subject().CN = "dockerapi"
-      cert.set_serial_number(int(uuid.uuid4()))
-      cert.gmtime_adj_notBefore(0)
-      cert.gmtime_adj_notAfter(10*365*24*60*60)
-      cert.set_issuer(cert.get_subject())
-      cert.set_pubkey(pkey)
-      cert.sign(pkey, 'sha512')
-      cert = crypto.dump_certificate(crypto.FILETYPE_PEM, cert)
-      pkey = crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey)
-      with os.fdopen(os.open('/cert.pem', os.O_WRONLY | os.O_CREAT, 0o644), 'w') as handle:
-        handle.write(cert)
-      with os.fdopen(os.open('/key.pem', os.O_WRONLY | os.O_CREAT, 0o600), 'w') as handle:
-        handle.write(pkey)
-      success = True
-    except:
-      time.sleep(1)
-      try:
-        os.remove('/cert.pem')
-        os.remove('/key.pem')
-      except OSError:
-        pass
-
 api.add_resource(containers_get, '/containers/json')
-api.add_resource(container_get, '/containers/<string:container_id>/json')
+api.add_resource(container_get,  '/containers/<string:container_id>/json')
 api.add_resource(container_post, '/containers/<string:container_id>/<string:post_action>')
 
 if __name__ == '__main__':
@@ -397,5 +397,4 @@ if __name__ == '__main__':
     time.sleep(1)
     if killer.kill_now:
       break
-  print "Stopping dockerapi-mailcow"
+  print ("Stopping dockerapi-mailcow")
-

data/Dockerfiles/dovecot/Dockerfile

@@ -3,117 +3,112 @@ LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
 ARG DEBIAN_FRONTEND=noninteractive
 ENV LC_ALL C
-ENV DOVECOT_VERSION 2.3.4
-ENV PIGEONHOLE_VERSION 0.5.4
 
-RUN apt-get update && apt-get -y --no-install-recommends install \
+# Add groups and users before installing Dovecot to not break compatibility
-  automake \
+RUN groupadd -g 5000 vmail \
-  autotools-dev \
-  build-essential \
-  ca-certificates \
-  cpanminus \
-  curl \
-  default-libmysqlclient-dev \
-  dnsutils \
-  gettext \
-  jq \
-  libjson-webtoken-perl \
-  libcgi-pm-perl \
-  libcrypt-openssl-rsa-perl \
-  libdata-uniqid-perl \
-  libhtml-parser-perl \
-  libmail-imapclient-perl \
-  libparse-recdescent-perl \
-  libsys-meminfo-perl \
-  libtest-mockobject-perl \
-  libwww-perl \
-  libauthen-ntlm-perl \
-  libbz2-dev \
-  libcrypt-ssleay-perl \
-  libcurl4-openssl-dev \
-  libdbd-mysql-perl \
-  libdbi-perl \
-  libdigest-hmac-perl \
-  libexpat1-dev \
-  libfile-copy-recursive-perl \
-  libio-compress-perl \
-  libio-socket-inet6-perl \
-  libio-socket-ssl-perl \
-  libio-tee-perl \
-  libipc-run-perl \
-  libldap2-dev \
-  liblockfile-simple-perl \
-  liblz-dev \
-  liblz4-dev \
-  liblzma-dev \
-  libmodule-scandeps-perl \
-  libnet-ssleay-perl \
-  libpam-dev \
-  libpar-packer-perl \
-  libreadonly-perl \
-  libssl-dev \
-  libterm-readkey-perl \
-  libtest-pod-perl \
-  libtest-simple-perl \
-  libtry-tiny-perl \
-  libunicode-string-perl \
-  libproc-processtable-perl \
-  libtest-nowarnings-perl \
-  libtest-deep-perl \
-  libtest-warn-perl \
-  libregexp-common-perl \
-  liburi-perl \
-  lzma-dev \
-  python-html2text \
-  python-jinja2 \
-  python-mysql.connector \
-  python-redis \
-  make \
-  mysql-client \
-  procps \
-  supervisor \
-  cron \
-  redis-server \
-  syslog-ng \
-  syslog-ng-core \
-  syslog-ng-mod-redis \
-  && rm -rf /var/lib/apt/lists/* \
-  && curl https://www.dovecot.org/releases/2.3/dovecot-$DOVECOT_VERSION.tar.gz | tar xvz  \
-  && cd dovecot-$DOVECOT_VERSION \
-  && ./configure --with-solr --with-mysql --with-ldap --with-lzma --with-lz4 --with-ssl=openssl --with-notify=inotify --with-storages=mdbox,sdbox,maildir,mbox,imapc,pop3c --with-bzlib --with-zlib --enable-hardening \
-  && make -j3 \
-  && make install \
-  && make clean \
-  && cd .. && rm -rf dovecot-$DOVECOT_VERSION \
-  && curl https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-$PIGEONHOLE_VERSION.tar.gz | tar xvz  \
-  && cd dovecot-2.3-pigeonhole-$PIGEONHOLE_VERSION \
-  && ./configure \
-  && make -j3 \
-  && make install \
-  && make clean \
-  && cd .. \
-  && rm -rf dovecot-2.3-pigeonhole-$PIGEONHOLE_VERSION \
-  && cpanm Data::Uniqid Mail::IMAPClient String::Util \
-  && groupadd -g 5000 vmail \
   && groupadd -g 401 dovecot \
   && groupadd -g 402 dovenull \
   && useradd -g vmail -u 5000 vmail -d /var/vmail \
   && useradd -c "Dovecot unprivileged user" -d /dev/null -u 401 -g dovecot -s /bin/false dovecot \
   && useradd -c "Dovecot login user" -d /dev/null -u 402 -g dovenull -s /bin/false dovenull \
   && touch /etc/default/locale \
-  && apt-get purge -y build-essential automake autotools-dev default-libmysqlclient-dev libbz2-dev libcurl4-openssl-dev libexpat1-dev liblz-dev liblz4-dev liblzma-dev libpam-dev libssl-dev lzma-dev \
+  && apt-get update \
+  && apt-get -y --no-install-recommends install \
+  apt-transport-https \
+  ca-certificates \
+  cpanminus \
+  cron \
+  curl \
+  dnsutils \
+  dirmngr \
+  gettext \
+  gnupg2 \
+  jq \
+  libauthen-ntlm-perl \
+  libcgi-pm-perl \
+  libcrypt-openssl-rsa-perl \
+  libcrypt-ssleay-perl \
+  libdata-uniqid-perl \
+  libdbd-mysql-perl \
+  libdbi-perl \
+  libdigest-hmac-perl \
+  libdist-checkconflicts-perl \
+  libfile-copy-recursive-perl \
+  libfile-tail-perl \
+  libhtml-parser-perl \
+  libio-compress-perl \
+  libio-socket-inet6-perl \
+  libio-socket-ssl-perl \
+  libio-tee-perl \
+  libipc-run-perl \
+  libjson-webtoken-perl \
+  liblockfile-simple-perl \
+  libmail-imapclient-perl \
+  libmodule-implementation-perl \
+  libmodule-scandeps-perl \
+  libnet-ssleay-perl \
+  libpackage-stash-perl \
+  libpackage-stash-xs-perl \
+  libpar-packer-perl \
+  libparse-recdescent-perl \
+  libproc-processtable-perl \
+  libreadonly-perl \
+  libregexp-common-perl \
+  libsys-meminfo-perl \
+  libterm-readkey-perl \
+  libtest-deep-perl \
+  libtest-fatal-perl \
+  libtest-mock-guard-perl \
+  libtest-mockobject-perl \
+  libtest-nowarnings-perl \
+  libtest-pod-perl \
+  libtest-requires-perl \
+  libtest-simple-perl \
+  libtest-warn-perl \
+  libtry-tiny-perl \
+  libunicode-string-perl \
+  liburi-perl \
+  libwww-perl \
+  mysql-client \
+  procps \
+  python-html2text \
+  python-jinja2 \
+  python-mysql.connector \
+  python-redis \
+  redis-server \
+  supervisor \
+  syslog-ng \
+  syslog-ng-core \
+  syslog-ng-mod-redis \
+  && apt-key adv --fetch-keys https://repo.dovecot.org/DOVECOT-REPO-GPG \
+  && echo 'deb https://repo.dovecot.org/ce-2.3-latest/debian/stretch stretch main' > /etc/apt/sources.list.d/dovecot.list \
+  && apt-get update \
+  && apt-get -y --no-install-recommends install \
+  dovecot-lua \
+  dovecot-managesieved \
+  dovecot-sieve \
+  dovecot-lmtpd \
+  dovecot-ldap \
+  dovecot-mysql \
+  dovecot-core \
+  dovecot-pop3d \
+  dovecot-imapd \
+  dovecot-solr \
   && apt-get autoremove --purge -y \
-  && rm -rf /tmp/* /var/tmp/*
+  && apt-get autoclean \
+  && rm -rf /var/lib/apt/lists/* \
+  && rm -rf /tmp/* /var/tmp/* /etc/cron.daily/*
 
 COPY trim_logs.sh /usr/local/bin/trim_logs.sh
+COPY clean_q_aged.sh /usr/local/bin/clean_q_aged.sh
 COPY syslog-ng.conf /etc/syslog-ng/syslog-ng.conf
 COPY imapsync /usr/local/bin/imapsync
 COPY postlogin.sh /usr/local/bin/postlogin.sh
 COPY imapsync_cron.pl /usr/local/bin/imapsync_cron.pl
-COPY report-spam.sieve /usr/local/lib/dovecot/sieve/report-spam.sieve
+COPY report-spam.sieve /usr/lib/dovecot/sieve/report-spam.sieve
-COPY report-ham.sieve /usr/local/lib/dovecot/sieve/report-ham.sieve
+COPY report-ham.sieve /usr/lib/dovecot/sieve/report-ham.sieve
-COPY rspamd-pipe-ham /usr/local/lib/dovecot/sieve/rspamd-pipe-ham
+COPY rspamd-pipe-ham /usr/lib/dovecot/sieve/rspamd-pipe-ham
-COPY rspamd-pipe-spam /usr/local/lib/dovecot/sieve/rspamd-pipe-spam
+COPY rspamd-pipe-spam /usr/lib/dovecot/sieve/rspamd-pipe-spam
 COPY sa-rules.sh /usr/local/bin/sa-rules.sh
 COPY maildir_gc.sh /usr/local/bin/maildir_gc.sh
 COPY docker-entrypoint.sh /

data/Dockerfiles/dovecot/clean_q_aged.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+MAX_AGE=$(redis-cli --raw -h redis-mailcow GET Q_MAX_AGE)
+
+if [[ -z ${MAX_AGE} ]]; then
+  echo "Max age for quarantine items not defined"
+  exit 1
+fi
+
+NUM_REGEXP='^[0-9]+$'
+if ! [[ ${MAX_AGE} =~ ${NUM_REGEXP} ]] ; then
+  echo "Max age for quarantine items invalid"
+  exit 1
+fi
+
+TO_DELETE=$(mysql --socket=/var/run/mysqld/mysqld.sock -u __DBUSER__ -p__DBPASS__ __DBNAME__ -e "SELECT COUNT(id) FROM quarantine WHERE created < NOW() - INTERVAL ${MAX_AGE//[!0-9]/} DAY" -BN)
+mysql --socket=/var/run/mysqld/mysqld.sock -u __DBUSER__ -p__DBPASS__ __DBNAME__ -e "DELETE FROM quarantine WHERE created < NOW() - INTERVAL ${MAX_AGE//[!0-9]/} DAY"
+echo "Deleted ${TO_DELETE} items from quarantine table (max age is ${MAX_AGE//[!0-9]/} days)"

data/Dockerfiles/dovecot/docker-entrypoint.sh

@@ -16,10 +16,14 @@ sed -i "s/__DBUSER__/${DBUSER}/g" /usr/local/bin/quarantine_notify.py
 sed -i "s/__DBPASS__/${DBPASS}/g" /usr/local/bin/quarantine_notify.py
 sed -i "s/__DBNAME__/${DBNAME}/g" /usr/local/bin/quarantine_notify.py
 
+sed -i "s/__DBUSER__/${DBUSER}/g" /usr/local/bin/clean_q_aged.sh
+sed -i "s/__DBPASS__/${DBPASS}/g" /usr/local/bin/clean_q_aged.sh
+sed -i "s/__DBNAME__/${DBNAME}/g" /usr/local/bin/clean_q_aged.sh
+
 sed -i "s/__LOG_LINES__/${LOG_LINES}/g" /usr/local/bin/trim_logs.sh
 
 # Create missing directories
-[[ ! -d /usr/local/etc/dovecot/sql/ ]] && mkdir -p /usr/local/etc/dovecot/sql/
+[[ ! -d /etc/dovecot/sql/ ]] && mkdir -p /etc/dovecot/sql/
 [[ ! -d /var/vmail/_garbage ]] && mkdir -p /var/vmail/_garbage
 [[ ! -d /var/vmail/sieve ]] && mkdir -p /var/vmail/sieve
 [[ ! -d /etc/sogo ]] && mkdir -p /etc/sogo
@@ -29,7 +33,8 @@ sed -i "s/__LOG_LINES__/${LOG_LINES}/g" /usr/local/bin/trim_logs.sh
 DBPASS=$(echo ${DBPASS} | sed 's/"/\\"/g')
 
 # Create quota dict for Dovecot
-cat <<EOF > /usr/local/etc/dovecot/sql/dovecot-dict-sql-quota.conf
+cat <<EOF > /etc/dovecot/sql/dovecot-dict-sql-quota.conf
+# Autogenerated by mailcow
 connect = "host=/var/run/mysqld/mysqld.sock dbname=${DBNAME} user=${DBUSER} password=${DBPASS}"
 map {
   pattern = priv/quota/storage
@@ -46,7 +51,8 @@ map {
 EOF
 
 # Create dict used for sieve pre and postfilters
-cat <<EOF > /usr/local/etc/dovecot/sql/dovecot-dict-sql-sieve_before.conf
+cat <<EOF > /etc/dovecot/sql/dovecot-dict-sql-sieve_before.conf
+# Autogenerated by mailcow
 connect = "host=/var/run/mysqld/mysqld.sock dbname=${DBNAME} user=${DBUSER} password=${DBPASS}"
 map {
   pattern = priv/sieve/name/\$script_name
@@ -68,7 +74,8 @@ map {
 }
 EOF
 
-cat <<EOF > /usr/local/etc/dovecot/sql/dovecot-dict-sql-sieve_after.conf
+cat <<EOF > /etc/dovecot/sql/dovecot-dict-sql-sieve_after.conf
+# Autogenerated by mailcow
 connect = "host=/var/run/mysqld/mysqld.sock dbname=${DBNAME} user=${DBUSER} password=${DBPASS}"
 map {
   pattern = priv/sieve/name/\$script_name
@@ -90,36 +97,41 @@ map {
 }
 EOF
 
-echo -n ${ACL_ANYONE} > /usr/local/etc/dovecot/acl_anyone
+echo -n ${ACL_ANYONE} > /etc/dovecot/acl_anyone
 
 if [[ "${SKIP_SOLR}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
-echo -n 'quota acl zlib listescape mail_crypt mail_crypt_acl mail_log notify' > /usr/local/etc/dovecot/mail_plugins
+echo -n 'quota acl zlib listescape mail_crypt mail_crypt_acl mail_log notify' > /etc/dovecot/mail_plugins
-echo -n 'quota imap_quota imap_acl acl zlib imap_zlib imap_sieve listescape mail_crypt mail_crypt_acl notify mail_log' > /usr/local/etc/dovecot/mail_plugins_imap
+echo -n 'quota imap_quota imap_acl acl zlib imap_zlib imap_sieve listescape mail_crypt mail_crypt_acl notify mail_log' > /etc/dovecot/mail_plugins_imap
-echo -n 'quota sieve acl zlib listescape mail_crypt mail_crypt_acl' > /usr/local/etc/dovecot/mail_plugins_lmtp
+echo -n 'quota sieve acl zlib listescape mail_crypt mail_crypt_acl' > /etc/dovecot/mail_plugins_lmtp
 else
-echo -n 'quota acl zlib listescape mail_crypt mail_crypt_acl mail_log notify fts fts_solr' > /usr/local/etc/dovecot/mail_plugins
+echo -n 'quota acl zlib listescape mail_crypt mail_crypt_acl mail_log notify fts fts_solr' > /etc/dovecot/mail_plugins
-echo -n 'quota imap_quota imap_acl acl zlib imap_zlib imap_sieve listescape mail_crypt mail_crypt_acl notify mail_log fts fts_solr' > /usr/local/etc/dovecot/mail_plugins_imap
+echo -n 'quota imap_quota imap_acl acl zlib imap_zlib imap_sieve listescape mail_crypt mail_crypt_acl notify mail_log fts fts_solr' > /etc/dovecot/mail_plugins_imap
-echo -n 'quota sieve acl zlib listescape mail_crypt mail_crypt_acl fts fts_solr' > /usr/local/etc/dovecot/mail_plugins_lmtp
+echo -n 'quota sieve acl zlib listescape mail_crypt mail_crypt_acl fts fts_solr' > /etc/dovecot/mail_plugins_lmtp
 fi
-chmod 644 /usr/local/etc/dovecot/mail_plugins /usr/local/etc/dovecot/mail_plugins_imap /usr/local/etc/dovecot/mail_plugins_lmtp /templates/quarantine.tpl
+chmod 644 /etc/dovecot/mail_plugins /etc/dovecot/mail_plugins_imap /etc/dovecot/mail_plugins_lmtp /templates/quarantine.tpl
 
-cat <<EOF > /usr/local/etc/dovecot/sql/dovecot-dict-sql-userdb.conf
+cat <<EOF > /etc/dovecot/sql/dovecot-dict-sql-userdb.conf
+# Autogenerated by mailcow
 driver = mysql
 connect = "host=/var/run/mysqld/mysqld.sock dbname=${DBNAME} user=${DBUSER} password=${DBPASS}"
-user_query = SELECT CONCAT(JSON_UNQUOTE(JSON_EXTRACT(attributes, '$.mailbox_format')), mailbox_path_prefix, '%d/%n/:VOLATILEDIR=/var/volatile/%u') AS mail, 5000 AS uid, 5000 AS gid, concat('*:bytes=', quota) AS quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
+user_query = SELECT CONCAT(JSON_UNQUOTE(JSON_EXTRACT(attributes, '$.mailbox_format')), mailbox_path_prefix, '%d/%n/${MAILDIR_SUB}:VOLATILEDIR=/var/volatile/%u') AS mail, 5000 AS uid, 5000 AS gid, concat('*:bytes=', quota) AS quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
 iterate_query = SELECT username FROM mailbox WHERE active='1';
 EOF
 
 # Create pass dict for Dovecot
-cat <<EOF > /usr/local/etc/dovecot/sql/dovecot-dict-sql-passdb.conf
+cat <<EOF > /etc/dovecot/sql/dovecot-dict-sql-passdb.conf
+# Autogenerated by mailcow
 driver = mysql
 connect = "host=/var/run/mysqld/mysqld.sock dbname=${DBNAME} user=${DBUSER} password=${DBPASS}"
 default_pass_scheme = SSHA256
 password_query = SELECT password FROM mailbox WHERE active = '1' AND username = '%u' AND domain IN (SELECT domain FROM domain WHERE domain='%d' AND active='1') AND JSON_EXTRACT(attributes, '$.force_pw_update') NOT LIKE '%%1%%'
 EOF
 
-# Create global sieve_after script
+# Migrate old sieve_after file
-cat /usr/local/etc/dovecot/sieve_after > /var/vmail/sieve/global.sieve
+[[ -f /etc/dovecot/sieve_after ]] && mv /etc/dovecot/sieve_after /etc/dovecot/global_sieve_after
+# Create global sieve scripts
+cat /etc/dovecot/global_sieve_after > /var/vmail/sieve/global_sieve_after.sieve
+cat /etc/dovecot/global_sieve_before > /var/vmail/sieve/global_sieve_before.sieve
 
 # Check permissions of vmail/attachments directory.
 # Do not do this every start-up, it may take a very long time. So we use a stat check here.
@@ -127,14 +139,51 @@ if [[ $(stat -c %U /var/vmail/) != "vmail" ]] ; then chown -R vmail:vmail /var/v
 if [[ $(stat -c %U /var/vmail/_garbage) != "vmail" ]] ; then chown -R vmail:vmail /var/vmail/_garbage ; fi
 if [[ $(stat -c %U /var/attachments) != "vmail" ]] ; then chown -R vmail:vmail /var/attachments ; fi
 
+# Cleanup random user maildirs
+rm -rf /var/vmail/mailcow.local/*
+
+
 # Create random master for SOGo sieve features
 RAND_USER=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 16 | head -n 1)
 RAND_PASS=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 24 | head -n 1)
 
-echo ${RAND_USER}@mailcow.local:{SHA1}$(echo -n ${RAND_PASS} | sha1sum | awk '{print $1}') > /usr/local/etc/dovecot/dovecot-master.passwd
+echo ${RAND_USER}@mailcow.local:{SHA1}$(echo -n ${RAND_PASS} | sha1sum | awk '{print $1}') > /etc/dovecot/dovecot-master.passwd
-echo ${RAND_USER}@mailcow.local::5000:5000:::: > /usr/local/etc/dovecot/dovecot-master.userdb
+echo ${RAND_USER}@mailcow.local::5000:5000:::: > /etc/dovecot/dovecot-master.userdb
 echo ${RAND_USER}@mailcow.local:${RAND_PASS} > /etc/sogo/sieve.creds
 
+if [[ -z ${MAILDIR_SUB} ]]; then
+  MAILDIR_SUB_SHARED=
+else
+  MAILDIR_SUB_SHARED=/${MAILDIR_SUB}
+fi
+cat <<EOF > /etc/dovecot/shared_namespace.conf
+# Autogenerated by mailcow
+namespace {
+    type = shared
+    separator = /
+    prefix = Shared/%%u/
+    location = maildir:%%h${MAILDIR_SUB_SHARED}:INDEX=~${MAILDIR_SUB_SHARED}/Shared/%%u;CONTROL=~${MAILDIR_SUB_SHARED}/Shared/%%u
+    subscriptions = no
+    list = children
+}
+EOF
+
+if [[ "${ALLOW_ADMIN_EMAIL_LOGIN}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+    # Create random master Password for SOGo 'login as user' via proxy auth
+    RAND_PASS=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 32 | head -n 1)
+    echo -n ${RAND_PASS} > /etc/phpfpm/sogo-sso.pass
+    cat <<EOF > /etc/dovecot/sogo-sso.conf
+# Autogenerated by mailcow
+passdb {
+  driver = static
+  args = allow_real_nets=${IPV4_NETWORK}.248/32 password={plain}${RAND_PASS}
+}
+EOF
+else
+    rm -f /etc/dovecot/sogo-sso.pass
+    rm -f /etc/dovecot/sogo-sso.conf
+fi
+
 # 401 is user dovecot
 if [[ ! -s /mail_crypt/ecprivkey.pem || ! -s /mail_crypt/ecpubkey.pem ]]; then
 	openssl ecparam -name prime256v1 -genkey | openssl pkey -out /mail_crypt/ecprivkey.pem
@@ -145,43 +194,46 @@ else
 fi
 
 # Compile sieve scripts
-sievec /var/vmail/sieve/global.sieve
+sievec /var/vmail/sieve/global_sieve_before.sieve
-sievec /usr/local/lib/dovecot/sieve/report-spam.sieve
+sievec /var/vmail/sieve/global_sieve_after.sieve
-sievec /usr/local/lib/dovecot/sieve/report-ham.sieve
+sievec /usr/lib/dovecot/sieve/report-spam.sieve
+sievec /usr/lib/dovecot/sieve/report-ham.sieve
 
 # Fix permissions
-chown root:root /usr/local/etc/dovecot/sql/*.conf
+chown root:root /etc/dovecot/sql/*.conf
-chown root:dovecot /usr/local/etc/dovecot/sql/dovecot-dict-sql-sieve* /usr/local/etc/dovecot/sql/dovecot-dict-sql-quota*
+chown root:dovecot /etc/dovecot/sql/dovecot-dict-sql-sieve* /etc/dovecot/sql/dovecot-dict-sql-quota*
-chmod 640 /usr/local/etc/dovecot/sql/*.conf
+chmod 640 /etc/dovecot/sql/*.conf
 chown -R vmail:vmail /var/vmail/sieve
 chown -R vmail:vmail /var/volatile
 adduser vmail tty
 chmod g+rw /dev/console
-chmod +x /usr/local/lib/dovecot/sieve/rspamd-pipe-ham \
+chown root:tty /dev/console
-  /usr/local/lib/dovecot/sieve/rspamd-pipe-spam \
+chmod +x /usr/lib/dovecot/sieve/rspamd-pipe-ham \
+  /usr/lib/dovecot/sieve/rspamd-pipe-spam \
   /usr/local/bin/imapsync_cron.pl \
   /usr/local/bin/postlogin.sh \
   /usr/local/bin/imapsync \
   /usr/local/bin/trim_logs.sh \
   /usr/local/bin/sa-rules.sh \
+  /usr/local/bin/clean_q_aged.sh \
   /usr/local/bin/maildir_gc.sh \
   /usr/local/sbin/stop-supervisor.sh \
   /usr/local/bin/quota_notify.py
 
 # Setup cronjobs
 echo '* * * * *    root  /usr/local/bin/imapsync_cron.pl 2>&1 | /usr/bin/logger' > /etc/cron.d/imapsync
-echo '30 3 * * *   vmail /usr/local/bin/doveadm quota recalc -A' > /etc/cron.d/dovecot-sync
+#echo '30 3 * * *   vmail /usr/local/bin/doveadm quota recalc -A' > /etc/cron.d/dovecot-sync
 echo '* * * * *    vmail /usr/local/bin/trim_logs.sh >> /dev/console 2>&1' > /etc/cron.d/trim_logs
 echo '25 * * * *   vmail /usr/local/bin/maildir_gc.sh >> /dev/console 2>&1' > /etc/cron.d/maildir_gc
 echo '30 1 * * *   root  /usr/local/bin/sa-rules.sh  >> /dev/console 2>&1' > /etc/cron.d/sa-rules
-echo '0 2 * * *    root  /usr/bin/curl http://solr:8983/solr/dovecot/update?optimize=true >> /dev/console 2>&1' > /etc/cron.d/solr-optimize
+echo '0 2 * * *    root  /usr/bin/curl http://solr:8983/solr/dovecot-fts/update?optimize=true >> /dev/console 2>&1' > /etc/cron.d/solr-optimize
 echo '*/20 * * * * vmail /usr/local/bin/quarantine_notify.py >> /dev/console 2>&1' > /etc/cron.d/quarantine_notify
-
+echo '15 4 * * * vmail /usr/local/bin/clean_q_aged.sh >> /dev/console 2>&1' > /etc/cron.d/clean_q_aged
 # Fix more than 1 hardlink issue
 touch /etc/crontab /etc/cron.*/*
 
 # Clean old PID if any
-[[ -f /usr/local/var/run/dovecot/master.pid ]] && rm /usr/local/var/run/dovecot/master.pid
+[[ -f /var/run/dovecot/master.pid ]] && rm /var/run/dovecot/master.pid
 
 # Clean stopped imapsync jobs
 rm -f /tmp/imapsync_busy.lock
@@ -191,6 +243,20 @@ IMAPSYNC_TABLE=$(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBP
 # Envsubst maildir_gc
 echo "$(envsubst < /usr/local/bin/maildir_gc.sh)" > /usr/local/bin/maildir_gc.sh
 
+PUBKEY_MCRYPT=$(doveconf -P | grep -i mail_crypt_global_public_key | cut -d '<' -f2)
+if [ -f ${PUBKEY_MCRYPT} ]; then
+  GUID=$(cat <(echo ${MAILCOW_HOSTNAME}) /mail_crypt/ecpubkey.pem | sha256sum | cut -d ' ' -f1 | tr -cd "[a-fA-F0-9.:/] ")
+  if [ ${#GUID} -eq 64 ]; then
+    mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} << EOF
+REPLACE INTO versions (application, version) VALUES ("GUID", "${GUID}");
+EOF
+  else
+    mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} << EOF
+REPLACE INTO versions (application, version) VALUES ("GUID", "INVALID");
+EOF
+  fi
+fi
+
 # Collect SA rules once now
 /usr/local/bin/sa-rules.sh
 

data/Dockerfiles/dovecot/imapsync_cron.pl

@@ -5,11 +5,11 @@ use LockFile::Simple qw(lock trylock unlock);
 use Proc::ProcessTable;
 use Data::Dumper qw(Dumper);
 use IPC::Run 'run';
-use String::Util 'trim';
 use File::Temp;
 use Try::Tiny;
 use sigtrap 'handler' => \&sig_handler, qw(INT TERM KILL QUIT);
 
+sub trim { my $s = shift; $s =~ s/^\s+|\s+$//g; return $s };
 my $t = Proc::ProcessTable->new;
 my $imapsync_running = grep { $_->{cmndline} =~ /^\/usr\/bin\/perl \/usr\/local\/bin\/imapsync\s/ } @{$t->table};
 if ($imapsync_running eq 1)
@@ -19,11 +19,20 @@ if ($imapsync_running eq 1)
 }
 
 sub qqw($) {
-  my @values = split('(?=--)', $_[0]);
+  my @params = ();
+  my @values = split(/(?=--)/, $_[0]);
   foreach my $val (@values) {
+    my @tmpparam = split(/ /, $val, 2);
+    foreach my $tmpval (@tmpparam) {
+        if ($tmpval ne '') {
+          push @params, $tmpval;
+        }
+    }
+  }
+  foreach my $val (@params) {
     $val=trim($val);
   }
-  return @values
+  return @params;
 }
 
 $run_dir="/tmp";
@@ -101,10 +110,6 @@ while ($row = $sth->fetchrow_arrayref()) {
   $timeout1            = @$row[19];
   $timeout2            = @$row[20];
 
-  $is_running = $dbh->prepare("UPDATE imapsync SET is_running = 1 WHERE id = ?");
-  $is_running->bind_param( 1, ${id} );
-  $is_running->execute();
-
   if ($enc1 eq "TLS") { $enc1 = "--tls1"; } elsif ($enc1 eq "SSL") { $enc1 = "--ssl1"; } else { undef $enc1; }
 
   my $template = $run_dir . '/imapsync.XXXXXXX';
@@ -118,43 +123,53 @@ while ($row = $sth->fetchrow_arrayref()) {
   my $custom_params_ref = \@custom_params_a;
 
   my $generated_cmds = [ "/usr/local/bin/imapsync",
-	"--tmpdir", "/tmp",
+  "--tmpdir", "/tmp",
-	"--nofoldersizes",
+  "--nofoldersizes",
-	($timeout1 gt "0" ? () : ('--timeout1', $timeout1)),
+  ($timeout1 gt "0" ? () : ('--timeout1', $timeout1)),
-	($timeout2 gt "0" ? () : ('--timeout2', $timeout2)),
+  ($timeout2 gt "0" ? () : ('--timeout2', $timeout2)),
-	($exclude eq ""	? () : ("--exclude", $exclude)),
+  ($exclude eq "" ? () : ("--exclude", $exclude)),
-	($subfolder2 eq "" ? () : ('--subfolder2', $subfolder2)),
+  ($subfolder2 eq "" ? () : ('--subfolder2', $subfolder2)),
-	($maxage eq "0" ? () : ('--maxage', $maxage)),
+  ($maxage eq "0" ? () : ('--maxage', $maxage)),
-	($maxbytespersecond eq "0" ? () : ('--maxbytespersecond', $maxbytespersecond)),
+  ($maxbytespersecond eq "0" ? () : ('--maxbytespersecond', $maxbytespersecond)),
-	($delete2duplicates	ne "1" ? () : ('--delete2duplicates')),
+  ($delete2duplicates ne "1" ? () : ('--delete2duplicates')),
-	($subscribeall	ne "1" ? () : ('--subscribeall')),
+  ($subscribeall  ne "1" ? () : ('--subscribeall')),
-	($delete1	ne "1" ? () : ('--delete')),
+  ($delete1 ne "1" ? () : ('--delete')),
   ($delete2 ne "1" ? () : ('--delete2')),
   ($automap ne "1" ? () : ('--automap')),
   ($skipcrossduplicates ne "1" ? () : ('--skipcrossduplicates')),
-	(!defined($enc1) ? () : ($enc1)),
+  (!defined($enc1) ? () : ($enc1)),
-	"--host1", $host1,
+  "--host1", $host1,
-	"--user1", $user1,
+  "--user1", $user1,
-	"--passfile1", $passfile1->filename,
+  "--passfile1", $passfile1->filename,
-	"--port1", $port1,
+  "--port1", $port1,
-	"--host2", "localhost",
+  "--host2", "localhost",
-	"--user2", $user2 . '*' . trim($master_user),
+  "--user2", $user2 . '*' . trim($master_user),
-	"--passfile2", $passfile2->filename,
+  "--passfile2", $passfile2->filename,
-	'--no-modulesversion'];
+  '--no-modulesversion',
+  '--noreleasecheck'];
 
   try {
+    $is_running = $dbh->prepare("UPDATE imapsync SET is_running = 1 WHERE id = ?");
+    $is_running->bind_param( 1, ${id} );
+    $is_running->execute();
+    
     run [@$generated_cmds, @$custom_params_ref], '&>', \my $stdout;
-    $update = $dbh->prepare("UPDATE imapsync SET returned_text = ?, last_run = NOW(), is_running = 0 WHERE id = ?");
+    
+    $update = $dbh->prepare("UPDATE imapsync SET returned_text = ? WHERE id = ?");
     $update->bind_param( 1, ${stdout} );
     $update->bind_param( 2, ${id} );
     $update->execute();
   } catch {
-    $update = $dbh->prepare("UPDATE imapsync SET returned_text = 'Could not start or finish imapsync', last_run = NOW(), is_running = 0 WHERE id = ?");
+    $update = $dbh->prepare("UPDATE imapsync SET returned_text = 'Could not start or finish imapsync' WHERE id = ?");
+    $update->bind_param( 1, ${id} );
+    $update->execute();
+  } finally {
+    $update = $dbh->prepare("UPDATE imapsync SET last_run = NOW(), is_running = 0 WHERE id = ?");
     $update->bind_param( 1, ${id} );
     $update->execute();
-    $lockmgr->unlock($lock_file);
   };
 
+
 }
 
 $sth->finish();

data/Dockerfiles/dovecot/quarantine_notify.py

@@ -83,13 +83,14 @@ def notify_rcpt(rcpt, msg_count, quarantine_acl):
       msg.attach(html_part)
       msg['To'] = str(rcpt)
       text = msg.as_string()
-      server.sendmail(msg['From'], msg['To'], text)
+      server.sendmail(msg['From'].encode("ascii", errors="ignore"), msg['To'], text)
       server.quit()
       for res in meta_query:
         query_mysql('UPDATE quarantine SET notified = 1 WHERE id = "%d"' % (res['id']), update = True)
       r.hset('Q_LAST_NOTIFIED', record['rcpt'], time_now)
       break
     except Exception as ex:
+      server.quit()
       print '%s'  % (ex)
       time.sleep(3)
 

data/Dockerfiles/dovecot/quota_notify.py

@@ -54,7 +54,7 @@ try:
   msg.attach(text_part)
   msg.attach(html_part)
   msg['To'] = username
-  p = Popen(['/usr/local/libexec/dovecot/dovecot-lda', '-d', username, '-o', '"plugin/quota=maildir:User quota:noenforcing"'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
+  p = Popen(['/usr/lib/dovecot/dovecot-lda', '-d', username, '-o', '"plugin/quota=maildir:User quota:noenforcing"'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
   p.communicate(input=msg.as_string())
 
 except Exception as ex:

data/Dockerfiles/dovecot/sa-rules.sh

@@ -1,25 +1,41 @@
 #!/bin/bash
+
+# Create temp directories
+[[ ! -d /tmp/sa-rules-schaal ]] && mkdir -p /tmp/sa-rules-schaal
 [[ ! -d /tmp/sa-rules-heinlein ]] && mkdir -p /tmp/sa-rules-heinlein
-if [[ ! -f /etc/rspamd/custom/sa-rules-heinlein ]]; then
+
+# Hash current SA rules
+if [[ ! -f /etc/rspamd/custom/sa-rules ]]; then
   HASH_SA_RULES=0
 else
-  HASH_SA_RULES=$(cat /etc/rspamd/custom/sa-rules-heinlein | md5sum | cut -d' ' -f1)
+  HASH_SA_RULES=$(cat /etc/rspamd/custom/sa-rules | md5sum | cut -d' ' -f1)
 fi
 
-curl --connect-timeout 15 --max-time 30 http://www.spamassassin.heinlein-support.de/$(dig txt 1.4.3.spamassassin.heinlein-support.de +short | tr -d '"').tar.gz --output /tmp/sa-rules.tar.gz
+# Deploy
-if [[ -f /tmp/sa-rules.tar.gz ]]; then
+## Heinlein
-  tar xfvz /tmp/sa-rules.tar.gz -C /tmp/sa-rules-heinlein
+curl --connect-timeout 15 --max-time 30 http://www.spamassassin.heinlein-support.de/$(dig txt 1.4.3.spamassassin.heinlein-support.de +short | tr -d '"').tar.gz --output /tmp/sa-rules-heinlein.tar.gz
-  # create complete list of rules in a single file
+if gzip -t /tmp/sa-rules-heinlein.tar.gz; then
-  cat /tmp/sa-rules-heinlein/*cf > /etc/rspamd/custom/sa-rules-heinlein
+  tar xfvz /tmp/sa-rules-heinlein.tar.gz -C /tmp/sa-rules-heinlein
-  # Only restart rspamd-mailcow when rules changed
+  cat /tmp/sa-rules-heinlein/*cf > /etc/rspamd/custom/sa-rules
-  if [[ $(cat /etc/rspamd/custom/sa-rules-heinlein | md5sum | cut -d' ' -f1) != ${HASH_SA_RULES} ]]; then
+fi
-    CONTAINER_NAME=rspamd-mailcow
+## Schaal
-    CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | \
+curl --connect-timeout 15 --max-time 30 http://sa.schaal-it.net/$(dig txt 1.4.3.sa.schaal-it.net +short | tr -d '"').tar.gz --output /tmp/sa-rules-schaal.tar.gz
-      jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | \
+if gzip -t /tmp/sa-rules-schaal.tar.gz; then
-      jq -rc "select( .name | tostring | contains(\"${CONTAINER_NAME}\")) | .id")
+  tar xfvz /tmp/sa-rules-schaal.tar.gz -C /tmp/sa-rules-schaal
-    if [[ ! -z ${CONTAINER_ID} ]]; then
+  # Append, do not overwrite
-      curl --silent --insecure -XPOST --connect-timeout 15 --max-time 120 https://dockerapi/containers/${CONTAINER_ID}/restart
+  cat /tmp/sa-rules-schaal/*cf >> /etc/rspamd/custom/sa-rules
-    fi
+fi
+
+if [[ "$(cat /etc/rspamd/custom/sa-rules | md5sum | cut -d' ' -f1)" != "${HASH_SA_RULES}" ]]; then
+  CONTAINER_NAME=rspamd-mailcow
+  CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | \
+    jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | \
+    jq -rc "select( .name | tostring | contains(\"${CONTAINER_NAME}\")) | .id")
+  if [[ ! -z ${CONTAINER_ID} ]]; then
+    curl --silent --insecure -XPOST --connect-timeout 15 --max-time 120 https://dockerapi/containers/${CONTAINER_ID}/restart
   fi
 fi
-rm -rf /tmp/sa-rules-heinlein /tmp/sa-rules.tar.gz
+
+# Cleanup
+rm -rf /tmp/sa-rules-heinlein /tmp/sa-rules-heinlein.tar.gz
+rm -rf /tmp/sa-rules-schaal /tmp/sa-rules-schaal.tar.gz

data/Dockerfiles/dovecot/supervisord.conf

@@ -12,7 +12,7 @@ stderr_logfile_maxbytes=0
 autostart=true
 
 [program:dovecot]
-command=/usr/local/sbin/dovecot -F
+command=/usr/sbin/dovecot -F
 autorestart=true
 
 [program:cron]

data/Dockerfiles/dovecot/syslog-ng.conf

@@ -31,10 +31,10 @@ destination d_redis_f2b_channel {
   );
 };
 filter f_mail { facility(mail); };
-filter f_not_watchdog { not message("172\.22\.1\.248"); };
+#filter f_not_watchdog { not message("172\.22\.1\.248"); };
 log {
   source(s_src);
-  filter(f_not_watchdog);
+#  filter(f_not_watchdog);
   destination(d_stdout);
   filter(f_mail);
   destination(d_redis_ui_log);

data/Dockerfiles/dovecot/trim_logs.sh

@@ -15,4 +15,4 @@ catch_non_zero "/usr/bin/redis-cli -h redis LTRIM NETFILTER_LOG 0 __LOG_LINES__"
 catch_non_zero "/usr/bin/redis-cli -h redis LTRIM AUTODISCOVER_LOG 0 __LOG_LINES__"
 catch_non_zero "/usr/bin/redis-cli -h redis LTRIM API_LOG 0 __LOG_LINES__"
 catch_non_zero "/usr/bin/redis-cli -h redis LTRIM RL_LOG 0 __LOG_LINES__"
-
+catch_non_zero "/usr/bin/redis-cli -h redis LTRIM WATCHDOG_LOG 0 __LOG_LINES__"

data/Dockerfiles/netfilter/Dockerfile

@@ -1,13 +1,16 @@
-FROM alpine:3.9
+FROM alpine:3.10
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
 ENV XTABLES_LIBDIR /usr/lib/xtables
 ENV PYTHON_IPTABLES_XTABLES_VERSION 12
 ENV IPTABLES_LIBDIR /usr/lib
 
-RUN apk add -U python2 python-dev py-pip gcc musl-dev iptables ip6tables tzdata \
+RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories \
-  && pip2 install --upgrade python-iptables==0.13.0 redis ipaddress \
+  && apk add --virtual .build-deps gcc python3-dev libffi-dev openssl-dev \
-  && apk del python-dev py2-pip gcc
+  && apk add -U python3 iptables=1.6.2-r1 ip6tables=1.6.2-r1 tzdata musl-dev \
+  && pip3 install --upgrade pip python-iptables redis ipaddress dnspython \
+#  && pip3 install --upgrade pip python-iptables==0.13.0 redis ipaddress dnspython \
+  && apk del .build-deps
 
 COPY server.py /
-CMD ["python2", "-u", "/server.py"]
+CMD ["python3", "-u", "/server.py"]

data/Dockerfiles/netfilter/server.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 
 import re
 import os
@@ -6,19 +6,22 @@ import time
 import atexit
 import signal
 import ipaddress
+from collections import Counter
 from random import randint
 from threading import Thread
 from threading import Lock
 import redis
 import json
 import iptc
+import dns.resolver
+import dns.exception
 
 while True:
   try:
     r = redis.StrictRedis(host=os.getenv('IPV4_NETWORK', '172.22.1') + '.249', decode_responses=True, port=6379, db=0)
     r.ping()
   except Exception as ex:
-    print '%s - trying again in 3 seconds'  % (ex)
+    print('%s - trying again in 3 seconds'  % (ex))
     time.sleep(3)
   else:
     break
@@ -31,13 +34,34 @@ RULES[2] = '-login: Disconnected \(auth failed, .+\): user=.*, method=.+, rip=([
 RULES[3] = '-login: Aborted login \(tried to use disallowed .+\): user=.+, rip=([0-9a-f\.:]+), lip.+'
 RULES[4] = 'SOGo.+ Login from \'([0-9a-f\.:]+)\' for user .+ might not have worked'
 RULES[5] = 'mailcow UI: Invalid password for .+ by ([0-9a-f\.:]+)'
-#RULES[6] = '-login: Aborted login \(no auth .+\): user=.+, rip=([0-9a-f\.:]+), lip.+'
+RULES[6] = '([0-9a-f\.:]+) \"GET \/SOGo\/.* HTTP.+\" 403 .+'
+#RULES[7] = '-login: Aborted login \(no auth .+\): user=.+, rip=([0-9a-f\.:]+), lip.+'
+
+WHITELIST = []
+BLACKLIST= []
 
 bans = {}
-log = {}
+
 quit_now = False
 lock = Lock()
 
+def log(priority, message):
+  tolog = {}
+  tolog['time'] = int(round(time.time()))
+  tolog['priority'] = priority
+  tolog['message'] = message
+  r.lpush('NETFILTER_LOG', json.dumps(tolog, ensure_ascii=False))
+  print(message)
+  
+def logWarn(message):
+  log('warn', message)
+  
+def logCrit(message):
+  log('crit', message)
+  
+def logInfo(message):
+  log('info', message)
+  
 def refreshF2boptions():
   global f2boptions
   global quit_now
@@ -58,8 +82,8 @@ def refreshF2boptions():
     try:
       f2boptions = {}
       f2boptions = json.loads(r.get('F2B_OPTIONS'))
-    except ValueError, e:
+    except ValueError:
-      print 'Error loading F2B options: F2B_OPTIONS is not json'
+      print('Error loading F2B options: F2B_OPTIONS is not json')
       quit_now = True
 
 if r.exists('F2B_LOG'):
@@ -84,18 +108,10 @@ def mailcowChainOrder():
             if item.target.name == 'MAILCOW':
               target_found = True
               if position != 0:
-                log['time'] = int(round(time.time()))
+                logCrit('Error in %s chain order, restarting container' % (chain.name))
-                log['priority'] = 'crit'
-                log['message'] = 'Error in ' + chain.name + ' chain order, restarting container'
-                r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-                print log['message']
                 quit_now = True
           if not target_found:
-            log['time'] = int(round(time.time()))
+            logCrit('Error in %s chain: MAILCOW target not found, restarting container' % (chain.name))
-            log['priority'] = 'crit'
-            log['message'] = 'Error in ' + chain.name + ' chain: MAILCOW target not found, restarting container'
-            r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-            print log['message']
             quit_now = True
 
 def ban(address):
@@ -106,28 +122,28 @@ def ban(address):
   RETRY_WINDOW = int(f2boptions['retry_window'])
   NETBAN_IPV4 = '/' + str(f2boptions['netban_ipv4'])
   NETBAN_IPV6 = '/' + str(f2boptions['netban_ipv6'])
-  WHITELIST = r.hgetall('F2B_WHITELIST')
 
-  ip = ipaddress.ip_address(address.decode('ascii'))
+  ip = ipaddress.ip_address(address)
   if type(ip) is ipaddress.IPv6Address and ip.ipv4_mapped:
     ip = ip.ipv4_mapped
     address = str(ip)
   if ip.is_private or ip.is_loopback:
     return
 
-  self_network = ipaddress.ip_network(address.decode('ascii'))
+  self_network = ipaddress.ip_network(address)
-  if WHITELIST:
+  
-    for wl_key in WHITELIST:
+  with lock:
-      wl_net = ipaddress.ip_network(wl_key.decode('ascii'), False)
+    temp_whitelist = set(WHITELIST)
+
+  if temp_whitelist:
+    for wl_key in temp_whitelist:
+      wl_net = ipaddress.ip_network(wl_key, False)
+          
       if wl_net.overlaps(self_network):
-        log['time'] = int(round(time.time()))
+        logInfo('Address %s is whitelisted by rule %s' % (self_network, wl_net))
-        log['priority'] = 'info'
-        log['message'] = 'Address %s is whitelisted by rule %s' % (self_network, wl_net)
-        r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-        print 'Address %s is whitelisted by rule %s' % (self_network, wl_net)
         return
 
-  net = ipaddress.ip_network((address + (NETBAN_IPV4 if type(ip) is ipaddress.IPv4Address else NETBAN_IPV6)).decode('ascii'), strict=False)
+  net = ipaddress.ip_network((address + (NETBAN_IPV4 if type(ip) is ipaddress.IPv4Address else NETBAN_IPV6)), strict=False)
   net = str(net)
 
   if not net in bans or time.time() - bans[net]['last_attempt'] > RETRY_WINDOW:
@@ -142,11 +158,8 @@ def ban(address):
   active_window = time.time() - bans[net]['last_attempt']
 
   if bans[net]['attempts'] >= MAX_ATTEMPTS:
-    log['time'] = int(round(time.time()))
+    cur_time = int(round(time.time()))
-    log['priority'] = 'crit'
+    logCrit('Banning %s for %d minutes' % (net, BAN_TIME / 60))
-    log['message'] = 'Banning %s' % net
-    r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-    print 'Banning %s for %d minutes' % (net, BAN_TIME / 60)
     if type(ip) is ipaddress.IPv4Address:
       with lock:
         chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), 'MAILCOW')
@@ -165,29 +178,18 @@ def ban(address):
         rule.target = target
         if rule not in chain.rules:
           chain.insert_rule(rule)
-    r.hset('F2B_ACTIVE_BANS', '%s' % net, log['time'] + BAN_TIME)
+    r.hset('F2B_ACTIVE_BANS', '%s' % net, cur_time + BAN_TIME)
   else:
-    log['time'] = int(round(time.time()))
+    logWarn('%d more attempts in the next %d seconds until %s is banned' % (MAX_ATTEMPTS - bans[net]['attempts'], RETRY_WINDOW, net))
-    log['priority'] = 'warn'
-    log['message'] = '%d more attempts in the next %d seconds until %s is banned' % (MAX_ATTEMPTS - bans[net]['attempts'], RETRY_WINDOW, net)
-    r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-    print '%d more attempts in the next %d seconds until %s is banned' % (MAX_ATTEMPTS - bans[net]['attempts'], RETRY_WINDOW, net)
 
 def unban(net):
   global lock
-  log['time'] = int(round(time.time())) 
-  log['priority'] = 'info'
-  r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
   if not net in bans:
-   log['message'] = '%s is not banned, skipping unban and deleting from queue (if any)' % net
+   logInfo('%s is not banned, skipping unban and deleting from queue (if any)' % net)
-   r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-   print '%s is not banned, skipping unban and deleting from queue (if any)' % net
    r.hdel('F2B_QUEUE_UNBAN', '%s' % net)
    return
-  log['message'] = 'Unbanning %s' % net
+  logInfo('Unbanning %s' % net)
-  r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
+  if type(ipaddress.ip_network(net)) is ipaddress.IPv4Network:
-  print 'Unbanning %s' % net
-  if type(ipaddress.ip_network(net.decode('ascii'))) is ipaddress.IPv4Network:
     with lock:
       chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), 'MAILCOW')
       rule = iptc.Rule()
@@ -210,17 +212,47 @@ def unban(net):
   if net in bans:
     del bans[net]
 
+def permBan(net, unban=False):
+  global lock
+  
+  if type(ipaddress.ip_network(net, strict=False)) is ipaddress.IPv4Network:
+    with lock:
+      chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), 'MAILCOW')
+      rule = iptc.Rule()
+      rule.src = net
+      target = iptc.Target(rule, "REJECT")
+      rule.target = target
+      if rule not in chain.rules and not unban:
+        logCrit('Add host/network %s to blacklist' % net)
+        chain.insert_rule(rule)
+        r.hset('F2B_PERM_BANS', '%s' % net, int(round(time.time()))) 
+      elif rule in chain.rules and unban:
+        logCrit('Remove host/network %s from blacklist' % net)
+        chain.delete_rule(rule)
+        r.hdel('F2B_PERM_BANS', '%s' % net)
+  else:
+    with lock:
+      chain = iptc.Chain(iptc.Table6(iptc.Table6.FILTER), 'MAILCOW')
+      rule = iptc.Rule6()
+      rule.src = net
+      target = iptc.Target(rule, "REJECT")
+      rule.target = target
+      if rule not in chain.rules and not unban:
+        logCrit('Add host/network %s to blacklist' % net)
+        chain.insert_rule(rule)
+        r.hset('F2B_PERM_BANS', '%s' % net, int(round(time.time()))) 
+      elif rule in chain.rules and unban:
+        logCrit('Remove host/network %s from blacklist' % net)
+        chain.delete_rule(rule)
+        r.hdel('F2B_PERM_BANS', '%s' % net)
+    
 def quit(signum, frame):
   global quit_now
   quit_now = True
 
 def clear():
   global lock
-  log['time'] = int(round(time.time()))
+  logInfo('Clearing all bans')
-  log['priority'] = 'info'
-  log['message'] = 'Clearing all bans'
-  r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-  print 'Clearing all bans'
   for net in bans.copy():
     unban(net)
   with lock:
@@ -249,28 +281,20 @@ def clear():
     pubsub.unsubscribe()
 
 def watch():
-  log['time'] = int(round(time.time()))
+  logInfo('Watching Redis channel F2B_CHANNEL')
-  log['priority'] = 'info'
-  log['message'] = 'Watching Redis channel F2B_CHANNEL'
-  r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
   pubsub.subscribe('F2B_CHANNEL')
-  print 'Subscribing to Redis channel F2B_CHANNEL'
 
   while not quit_now:
     for item in pubsub.listen():
-      for rule_id, rule_regex in RULES.iteritems():
+      for rule_id, rule_regex in RULES.items():
         if item['data'] and item['type'] == 'message':
           result = re.search(rule_regex, item['data'])
           if result:
             addr = result.group(1)
-            ip = ipaddress.ip_address(addr.decode('ascii'))
+            ip = ipaddress.ip_address(addr)
             if ip.is_private or ip.is_loopback:
               continue
-            print '%s matched rule id %d' % (addr, rule_id)
+            logWarn('%s matched rule id %d' % (addr, rule_id))
-            log['time'] = int(round(time.time()))
-            log['priority'] = 'warn'
-            log['message'] = '%s matched rule id %d' % (addr, rule_id)
-            r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
             ban(addr)
 
 def snat4(snat_target):
@@ -294,11 +318,7 @@ def snat4(snat_target):
         chain = iptc.Chain(table, 'POSTROUTING')
         table.autocommit = False
         if get_snat4_rule() not in chain.rules:
-          log['time'] = int(round(time.time()))
+          logCrit('Added POSTROUTING rule for source network %s to SNAT target %s' % (get_snat4_rule().src, snat_target))  
-          log['priority'] = 'info'
-          log['message'] = 'Added POSTROUTING rule for source network ' + get_snat4_rule().src + ' to SNAT target ' + snat_target
-          r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-          print log['message']
           chain.insert_rule(get_snat4_rule())
           table.commit()
         else:
@@ -309,7 +329,7 @@ def snat4(snat_target):
           table.commit()
         table.autocommit = True
       except:
-        print 'Error running SNAT4, retrying...' 
+        print('Error running SNAT4, retrying...') 
 
 def snat6(snat_target):
   global lock
@@ -332,11 +352,7 @@ def snat6(snat_target):
         chain = iptc.Chain(table, 'POSTROUTING')
         table.autocommit = False
         if get_snat6_rule() not in chain.rules:
-          log['time'] = int(round(time.time()))
+          logInfo('Added POSTROUTING rule for source network %s to SNAT target %s' % (get_snat6_rule().src, snat_target))
-          log['priority'] = 'info'
-          log['message'] = 'Added POSTROUTING rule for source network ' + get_snat6_rule().src + ' to SNAT target ' + snat_target
-          r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-          print log['message']
           chain.insert_rule(get_snat6_rule())
           table.commit()
         else:
@@ -347,14 +363,14 @@ def snat6(snat_target):
           table.commit()
         table.autocommit = True
       except:
-        print 'Error running SNAT6, retrying...' 
+        print('Error running SNAT6, retrying...') 
 
 def autopurge():
   while not quit_now:
     time.sleep(10)
     refreshF2boptions()
-    BAN_TIME = f2boptions['ban_time']
+    BAN_TIME = int(f2boptions['ban_time'])
-    MAX_ATTEMPTS = f2boptions['max_attempts']
+    MAX_ATTEMPTS = int(f2boptions['max_attempts'])
     QUEUE_UNBAN = r.hgetall('F2B_QUEUE_UNBAN')
     if QUEUE_UNBAN:
       for net in QUEUE_UNBAN:
@@ -364,9 +380,101 @@ def autopurge():
         if time.time() - bans[net]['last_attempt'] > BAN_TIME:
           unban(net)
 
+def isIpNetwork(address):
+  try:
+    ipaddress.ip_network(address, False)
+  except ValueError:
+    return False
+  return True
+
+          
+def genNetworkList(list):
+  resolver = dns.resolver.Resolver()
+  hostnames = []
+  networks = []
+
+  for key in list:
+    if isIpNetwork(key):
+      networks.append(key)
+    else:
+      hostnames.append(key)
+
+  for hostname in hostnames:
+    hostname_ips = []
+    for rdtype in ['A', 'AAAA']:
+      try:
+        answer = resolver.query(qname=hostname, rdtype=rdtype, lifetime=3)
+      except dns.exception.Timeout:
+        logInfo('Hostname %s timedout on resolve' % hostname)
+        break
+      except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
+        continue
+      except dns.exception.DNSException as dnsexception:
+        logInfo('%s' % dnsexception)
+        continue
+
+      for rdata in answer:
+        hostname_ips.append(rdata.to_text())
+
+    networks.extend(hostname_ips)
+      
+  return set(networks)
+
+def whitelistUpdate():
+  global lock
+  global quit_now
+  global WHITELIST
+  
+  while not quit_now:
+    start_time = time.time()
+    list = r.hgetall('F2B_WHITELIST')
+    
+    new_whitelist = []
+    
+    if list:
+      new_whitelist = genNetworkList(list)
+    
+    with lock:
+      if Counter(new_whitelist) != Counter(WHITELIST):
+        WHITELIST = new_whitelist
+        logInfo('Whitelist was changed, it has %s entries' % len(WHITELIST))
+
+    time.sleep(60.0 - ((time.time() - start_time) % 60.0)) 
+    
+def blacklistUpdate():
+  global quit_now
+  global BLACKLIST
+  
+  while not quit_now:
+    start_time = time.time()
+    list = r.hgetall('F2B_BLACKLIST')
+    
+    new_blacklist = []
+    
+    if list:
+      new_blacklist = genNetworkList(list)
+      
+    if Counter(new_blacklist) != Counter(BLACKLIST): 
+      addban = set(new_blacklist).difference(BLACKLIST)
+      delban = set(BLACKLIST).difference(new_blacklist)
+        
+      BLACKLIST = new_blacklist
+      logInfo('Blacklist was changed, it has %s entries' % len(BLACKLIST))
+        
+      if addban:
+        for net in addban:
+          permBan(net=net)
+            
+      if delban:
+        for net in delban:
+          permBan(net=net, unban=True)
+      
+        
+    time.sleep(60.0 - ((time.time() - start_time) % 60.0)) 
+      
 def initChain():
   # Is called before threads start, no locking
-  print "Initializing mailcow netfilter chain"
+  print("Initializing mailcow netfilter chain")
   # IPv4
   if not iptc.Chain(iptc.Table(iptc.Table.FILTER), "MAILCOW") in iptc.Table(iptc.Table.FILTER).chains:
     iptc.Table(iptc.Table.FILTER).create_chain("MAILCOW")
@@ -391,38 +499,7 @@ def initChain():
     rule.target = target
     if rule not in chain.rules:
       chain.insert_rule(rule)
-  # Apply blacklist
+ 
-  BLACKLIST = r.hgetall('F2B_BLACKLIST')
-  if BLACKLIST:
-    for bl_key in BLACKLIST:
-      if type(ipaddress.ip_network(bl_key.decode('ascii'), strict=False)) is ipaddress.IPv4Network:
-        chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), 'MAILCOW')
-        rule = iptc.Rule()
-        rule.src = bl_key
-        target = iptc.Target(rule, "REJECT")
-        rule.target = target
-        if rule not in chain.rules:
-          log['time'] = int(round(time.time()))
-          log['priority'] = 'crit'
-          log['message'] = 'Blacklisting host/network %s' % bl_key
-          r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-          print log['message']
-          chain.insert_rule(rule)
-          r.hset('F2B_PERM_BANS', '%s' % bl_key, int(round(time.time())))
-      else:
-        chain = iptc.Chain(iptc.Table6(iptc.Table6.FILTER), 'MAILCOW')
-        rule = iptc.Rule6()
-        rule.src = bl_key
-        target = iptc.Target(rule, "REJECT")
-        rule.target = target
-        if rule not in chain.rules:
-          log['time'] = int(round(time.time()))
-          log['priority'] = 'crit'
-          log['message'] = 'Blacklisting host/network %s' % bl_key
-          r.lpush('NETFILTER_LOG', json.dumps(log, ensure_ascii=False))
-          print log['message']
-          chain.insert_rule(rule)
-          r.hset('F2B_PERM_BANS', '%s' % bl_key, int(round(time.time())))
 
 if __name__ == '__main__':
 
@@ -437,25 +514,25 @@ if __name__ == '__main__':
 
   if os.getenv('SNAT_TO_SOURCE') and os.getenv('SNAT_TO_SOURCE') is not 'n':
     try:
-      snat_ip = os.getenv('SNAT_TO_SOURCE').decode('ascii')
+      snat_ip = os.getenv('SNAT_TO_SOURCE')
       snat_ipo = ipaddress.ip_address(snat_ip)
       if type(snat_ipo) is ipaddress.IPv4Address:
         snat4_thread = Thread(target=snat4,args=(snat_ip,))
         snat4_thread.daemon = True
         snat4_thread.start()
     except ValueError:
-      print os.getenv('SNAT_TO_SOURCE') + ' is not a valid IPv4 address'
+      print(os.getenv('SNAT_TO_SOURCE') + ' is not a valid IPv4 address')
 
   if os.getenv('SNAT6_TO_SOURCE') and os.getenv('SNAT6_TO_SOURCE') is not 'n':
     try:
-      snat_ip = os.getenv('SNAT6_TO_SOURCE').decode('ascii')
+      snat_ip = os.getenv('SNAT6_TO_SOURCE')
       snat_ipo = ipaddress.ip_address(snat_ip)
       if type(snat_ipo) is ipaddress.IPv6Address:
         snat6_thread = Thread(target=snat6,args=(snat_ip,))
         snat6_thread.daemon = True
         snat6_thread.start()
     except ValueError:
-      print os.getenv('SNAT6_TO_SOURCE') + ' is not a valid IPv6 address'
+      print(os.getenv('SNAT6_TO_SOURCE') + ' is not a valid IPv6 address')
 
   autopurge_thread = Thread(target=autopurge)
   autopurge_thread.daemon = True
@@ -465,6 +542,14 @@ if __name__ == '__main__':
   mailcowchainwatch_thread.daemon = True
   mailcowchainwatch_thread.start()
   
+  blacklistupdate_thread = Thread(target=blacklistUpdate)
+  blacklistupdate_thread.daemon = True
+  blacklistupdate_thread.start()
+
+  whitelistupdate_thread = Thread(target=whitelistUpdate)
+  whitelistupdate_thread.daemon = True
+  whitelistupdate_thread.start()
+
   signal.signal(signal.SIGTERM, quit)
   atexit.register(clear)
 

data/Dockerfiles/olefy/Dockerfile

@@ -0,0 +1,19 @@
+FROM alpine:3.10
+LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
+
+WORKDIR /app
+
+#RUN addgroup -S olefy && adduser -S olefy -G olefy \
+RUN apk add --virtual .build-deps gcc python3-dev musl-dev libffi-dev openssl-dev \
+  && apk add --update --no-cache python3 openssl tzdata libmagic \
+  && pip3 install --upgrade pip \
+  && pip3 install --upgrade oletools asyncio python-magic \
+  && apk del .build-deps
+
+ADD https://raw.githubusercontent.com/HeinleinSupport/olefy/master/olefy.py /app/
+
+RUN chown -R nobody:nobody /app /tmp
+
+USER nobody
+
+CMD ["python3", "-u", "/app/olefy.py"]

data/Dockerfiles/phpfpm/Dockerfile

@@ -1,11 +1,11 @@
-FROM php:7.3-fpm-alpine3.8
+FROM php:7.3-fpm-alpine3.10
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
-ENV APCU_PECL 5.1.16
+ENV APCU_PECL 5.1.17
-ENV IMAGICK_PECL 3.4.3
+ENV IMAGICK_PECL 3.4.4
 #ENV MAILPARSE_PECL 3.0.2
 ENV MEMCACHED_PECL 3.1.3
-ENV REDIS_PECL 4.2.0
+ENV REDIS_PECL 5.0.1
 
 RUN apk add -U --no-cache autoconf \
   bash \
@@ -53,13 +53,14 @@ RUN apk add -U --no-cache autoconf \
   && docker-php-ext-enable apcu imagick memcached mailparse redis \
   && pecl clear-cache \
   && docker-php-ext-configure intl \
+  && docker-php-ext-configure exif \
   && docker-php-ext-configure gd \
     --with-gd \
     --enable-gd-native-ttf \
     --with-freetype-dir=/usr/include/ \
     --with-png-dir=/usr/include/ \
     --with-jpeg-dir=/usr/include/ \
-  && docker-php-ext-install -j 4 gd gettext intl ldap opcache pcntl pdo pdo_mysql soap sockets xmlrpc zip \
+  && docker-php-ext-install -j 4 exif gd gettext intl ldap opcache pcntl pdo pdo_mysql soap sockets xmlrpc zip \
   && docker-php-ext-configure imap --with-imap --with-imap-ssl \
   && docker-php-ext-install -j 4 imap \
   && apk del --purge autoconf \

data/Dockerfiles/phpfpm/docker-entrypoint.sh

@@ -19,29 +19,48 @@ if [[ -z $(redis-cli --raw -h redis-mailcow GET Q_RELEASE_FORMAT) ]]; then
   redis-cli --raw -h redis-mailcow SET Q_RELEASE_FORMAT raw
 fi
 
+# Set max age of q items - if unset
+
+if [[ -z $(redis-cli --raw -h redis-mailcow GET Q_MAX_AGE) ]]; then
+  redis-cli --raw -h redis-mailcow SET Q_MAX_AGE 365
+fi
+
 # Check of mysql_upgrade
 
 CONTAINER_ID=
 # Todo: Better check if upgrade failed
 # This can happen due to a broken sogo_view
 [ -s /mysql_upgrade_loop ] && SQL_LOOP_C=$(cat /mysql_upgrade_loop)
-CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring | contains(\"mysql-mailcow\")) | .id")
+until [[ ! -z "${CONTAINER_ID}" ]] && [[ "${CONTAINER_ID}" =~ ^[[:alnum:]]*$ ]]; do
-if [[ ! -z "${CONTAINER_ID}" ]] && [[ "${CONTAINER_ID}" =~ [^a-zA-Z0-9] ]]; then
+  CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" 2> /dev/null | jq -rc "select( .name | tostring | contains(\"mysql-mailcow\")) | .id" 2> /dev/null)
-  SQL_UPGRADE_RETURN=$(curl --silent --insecure -XPOST https://dockerapi/containers/${CONTAINER_ID}/exec -d '{"cmd":"system", "task":"mysql_upgrade"}' --silent -H 'Content-type: application/json' | jq -r .type)
+done
-  if [[ ${SQL_UPGRADE_RETURN} == 'warning' ]]; then
+echo "MySQL @ ${CONTAINER_ID}"
-    if [ -z ${SQL_LOOP_C} ]; then
+SQL_UPGRADE_RETURN=$(curl --silent --insecure -XPOST https://dockerapi/containers/${CONTAINER_ID}/exec -d '{"cmd":"system", "task":"mysql_upgrade"}' --silent -H 'Content-type: application/json' | jq -r .type)
-      echo 1 > /mysql_upgrade_loop
+if [[ ${SQL_UPGRADE_RETURN} == 'warning' ]]; then
-      echo "MySQL applied an upgrade, restarting PHP-FPM..."
+  if [ -z ${SQL_LOOP_C} ]; then
-      exit 1
+    echo 1 > /mysql_upgrade_loop
+    echo "MySQL applied an upgrade"
+    POSTFIX=($(curl --silent --insecure https://dockerapi/containers/json | jq -r '.[] | {name: .Config.Labels["com.docker.compose.service"], id: .Id}' | jq -rc 'select( .name | tostring | contains("postfix-mailcow")) | .id' | tr "\n" " "))
+    if [[ -z ${POSTFIX} ]]; then
+      echo "Could not determine Postfix container ID, skipping Postfix restart."
     else
-      rm /mysql_upgrade_loop
+      echo "Restarting Postfix"
-      echo "MySQL was not applied previously, skipping. Restart php-fpm-mailcow to retry or run mysql_upgrade manually."
+      curl -X POST --silent --insecure https://dockerapi/containers/${POSTFIX}/restart | jq -r '.msg'
-      while ! mysqladmin status --socket=/var/run/mysqld/mysqld.sock -u${DBUSER} -p${DBPASS} --silent; do
+      echo "Sleeping 10 seconds..."
-        echo "Waiting for SQL to return..."
+      sleep 10
-        sleep 2
-      done
     fi
+    echo "Restarting PHP-FPM, bye"
+    exit 1
+  else
+    rm /mysql_upgrade_loop
+    echo "MySQL was not applied previously, skipping. Restart php-fpm-mailcow to retry or run mysql_upgrade manually."
+    while ! mysqladmin status --socket=/var/run/mysqld/mysqld.sock -u${DBUSER} -p${DBPASS} --silent; do
+      echo "Waiting for SQL to return..."
+      sleep 2
+    done
   fi
+else
+  echo "MySQL is up-to-date"
 fi
 
 # Trigger db init

data/Dockerfiles/postfix/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:bionic
+FROM debian:buster-slim
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
 ARG DEBIAN_FRONTEND=noninteractive
@@ -9,12 +9,17 @@ RUN dpkg-divert --local --rename --add /sbin/initctl \
 	&& dpkg-divert --local --rename --add /usr/bin/ischroot \
 	&& ln -sf /bin/true /usr/bin/ischroot
 
-RUN apt-get update && apt-get install -y --no-install-recommends \
+# Add groups and users before installing Postfix to not break compatibility
+RUN groupadd -g 102 postfix \
+  && groupadd -g 103 postdrop \
+  && useradd -g postfix -u 101 -d /var/spool/postfix -s /usr/sbin/nologin postfix \
+  && apt-get update && apt-get install -y --no-install-recommends \
 	ca-certificates \
 	curl \
 	dirmngr \
 	gnupg \
 	libsasl2-modules \
+  mariadb-client \
 	perl \
 	postfix \
 	postfix-mysql \
@@ -32,18 +37,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
   && printf '#!/bin/bash\n/usr/sbin/postconf -c /opt/postfix/conf "$@"' > /usr/local/sbin/postconf \
   && chmod +x /usr/local/sbin/postconf
 
-RUN addgroup --system --gid 600 zeyple \
-  && adduser --system --home /var/lib/zeyple --no-create-home --uid 600 --gid 600 --disabled-login zeyple \
-  && touch /var/log/zeyple.log \
-  && chown zeyple: /var/log/zeyple.log \
-  && mkdir -p /opt/mailman/var/data \
-  && touch /opt/mailman/var/data/postfix_lmtp \
-  && touch /opt/mailman/var/data/postfix_domains
-
-COPY zeyple.py /usr/local/bin/zeyple.py
-COPY zeyple.conf /etc/zeyple.conf
 COPY supervisord.conf /etc/supervisor/supervisord.conf
 COPY syslog-ng.conf /etc/syslog-ng/syslog-ng.conf
+COPY stop-supervisor.sh /usr/local/sbin/stop-supervisor.sh
 COPY postfix.sh /opt/postfix.sh
 COPY rspamd-pipe-ham /usr/local/bin/rspamd-pipe-ham
 COPY rspamd-pipe-spam /usr/local/bin/rspamd-pipe-spam

data/Dockerfiles/postfix/postfix.sh

@@ -4,14 +4,23 @@ trap "postfix stop" EXIT
 
 [[ ! -d /opt/postfix/conf/sql/ ]] && mkdir -p /opt/postfix/conf/sql/
 
+# Wait for MySQL to warm-up
+while ! mysqladmin status --socket=/var/run/mysqld/mysqld.sock -u${DBUSER} -p${DBPASS} --silent; do
+  echo "Waiting for database to come up..."
+  sleep 2
+done
+
 cat <<EOF > /etc/aliases
+# Autogenerated by mailcow
 null: /dev/null
+watchdog: /dev/null
 ham: "|/usr/local/bin/rspamd-pipe-ham"
 spam: "|/usr/local/bin/rspamd-pipe-spam"
 EOF
 newaliases;
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_relay_recipient_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -30,6 +39,7 @@ query = SELECT DISTINCT
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_tls_policy_override_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -38,6 +48,7 @@ query = SELECT CONCAT(policy, ' ', parameters) AS tls_policy FROM tls_policy_ove
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_tls_enforce_in_policy.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -55,6 +66,7 @@ query = SELECT IF(EXISTS(
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_sender_dependent_default_transport_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -86,6 +98,7 @@ query = SELECT GROUP_CONCAT(transport SEPARATOR '') AS transport_maps
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_transport_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -95,7 +108,18 @@ query = SELECT CONCAT('smtp_via_transport_maps:', nexthop) AS transport FROM tra
   AND destination = '%s';
 EOF
 
+cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_resource_maps.cf
+# Autogenerated by mailcow
+user = ${DBUSER}
+password = ${DBPASS}
+hosts = unix:/var/run/mysqld/mysqld.sock
+dbname = ${DBNAME}
+query = SELECT 'null@localhost' FROM mailbox
+  WHERE kind REGEXP 'location|thing|group' AND username = '%s';
+EOF
+
 cat <<EOF > /opt/postfix/conf/sql/mysql_sasl_passwd_maps_sender_dependent.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -104,8 +128,8 @@ query = SELECT CONCAT_WS(':', username, password) AS auth_data FROM relayhosts
   WHERE id IN (
     SELECT relayhost FROM domain
       WHERE CONCAT('@', domain) = '%s'
-      OR '%s' IN (
+      OR domain IN (
-        SELECT CONCAT('@', alias_domain) FROM alias_domain
+        SELECT target_domain FROM alias_domain WHERE CONCAT('@', alias_domain) =  '%s'
       )
   )
   AND active = '1'
@@ -113,6 +137,7 @@ query = SELECT CONCAT_WS(':', username, password) AS auth_data FROM relayhosts
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_sasl_passwd_maps_transport_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -124,18 +149,8 @@ query = SELECT CONCAT_WS(':', username, password) AS auth_data FROM transports
   LIMIT 1;
 EOF
 
-cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_alias_domain_catchall_maps.cf
-user = ${DBUSER}
-password = ${DBPASS}
-hosts = unix:/var/run/mysqld/mysqld.sock
-dbname = ${DBNAME}
-query = SELECT goto FROM alias, alias_domain
-  WHERE alias_domain.alias_domain = '%d'
-    AND alias.address = CONCAT('@', alias_domain.target_domain)
-    AND alias.active = 1 AND alias_domain.active='1'
-EOF
-
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_alias_domain_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -148,6 +163,7 @@ query = SELECT username FROM mailbox, alias_domain
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_alias_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -158,6 +174,7 @@ query = SELECT goto FROM alias
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_recipient_bcc_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -169,6 +186,7 @@ query = SELECT bcc_dest FROM bcc_maps
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_sender_bcc_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -180,6 +198,7 @@ query = SELECT bcc_dest FROM bcc_maps
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_recipient_canonical_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -190,6 +209,7 @@ query = SELECT new_dest FROM recipient_maps
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_domains_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -203,6 +223,7 @@ query = SELECT alias_domain from alias_domain WHERE alias_domain='%s' AND active
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_mailbox_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -211,6 +232,7 @@ query = SELECT CONCAT(JSON_UNQUOTE(JSON_EXTRACT(attributes, '$.mailbox_format'))
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_relay_domain_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -219,6 +241,7 @@ query = SELECT domain FROM domain WHERE domain='%s' AND backupmx = '1' AND activ
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_sender_acl.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -260,6 +283,7 @@ query = SELECT goto FROM alias
 EOF
 
 cat <<EOF > /opt/postfix/conf/sql/mysql_virtual_spamalias_maps.cf
+# Autogenerated by mailcow
 user = ${DBUSER}
 password = ${DBPASS}
 hosts = unix:/var/run/mysqld/mysqld.sock
@@ -269,10 +293,11 @@ query = SELECT goto FROM spamalias
     AND validity >= UNIX_TIMESTAMP()
 EOF
 
-# Reset GPG key permissions
+sed -i '/User overrides/q' /opt/postfix/conf/main.cf
-mkdir -p /var/lib/zeyple/keys
+echo >> /opt/postfix/conf/main.cf
-chmod 700 /var/lib/zeyple/keys
+if [ -f /opt/postfix/conf/extra.cf ]; then
-chown -R 600:600 /var/lib/zeyple/keys
+  cat /opt/postfix/conf/extra.cf >> /opt/postfix/conf/main.cf
+fi
 
 # Fix Postfix permissions
 chown -R root:postfix /opt/postfix/conf/sql/
@@ -282,7 +307,7 @@ chgrp -R postdrop /var/spool/postfix/maildrop
 postfix set-permissions
 
 # Check Postfix configuration
-postconf -c /opt/postfix/conf
+postconf -c /opt/postfix/conf > /dev/null
 
 if [[ $? != 0 ]]; then
   echo "Postfix configuration error, refusing to start."

data/Dockerfiles/postfix/supervisord.conf

@@ -1,4 +1,5 @@
 [supervisord]
+pidfile=/var/run/supervisord.pid
 nodaemon=true
 user=root
 
@@ -12,6 +13,10 @@ autostart=true
 
 [program:postfix]
 command=/opt/postfix.sh
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
 autorestart=true
 
 [eventlistener:processes]

data/Dockerfiles/postfix/syslog-ng.conf

@@ -1,9 +1,10 @@
-@version: 3.13
+@version: 3.19
 @include "scl.conf"
 options {
   chain_hostnames(off);
   flush_lines(0);
   use_dns(no);
+  dns_cache(no);
   use_fqdn(no);
   owner("root"); group("adm"); perm(0640);
   stats_freq(0);

data/Dockerfiles/postfix/zeyple.conf

@@ -1,9 +0,0 @@
-[zeyple]
-log_file = /dev/null
-
-[gpg]
-home = /var/lib/zeyple/keys
-
-[relay]
-host = localhost
-port = 10026

data/Dockerfiles/postfix/zeyple.py

@@ -1,274 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-import sys
-import os
-import logging
-import email
-import email.mime.multipart
-import email.mime.application
-import email.encoders
-import smtplib
-import copy
-from io import BytesIO
-
-try:
-    from configparser import SafeConfigParser  # Python 3
-except ImportError:
-    from ConfigParser import SafeConfigParser  # Python 2
-
-import gpgme
-
-# Boiler plate to avoid dependency on six
-# BBB: Python 2.7 support
-PY3K = sys.version_info > (3, 0)
-
-
-def message_from_binary(message):
-    if PY3K:
-        return email.message_from_bytes(message)
-    else:
-        return email.message_from_string(message)
-
-
-def as_binary_string(email):
-    if PY3K:
-        return email.as_bytes()
-    else:
-        return email.as_string()
-
-
-def encode_string(string):
-    if isinstance(string, bytes):
-        return string
-    else:
-        return string.encode('utf-8')
-
-
-__title__ = 'Zeyple'
-__version__ = '1.2.0'
-__author__ = 'Cédric Félizard'
-__license__ = 'AGPLv3+'
-__copyright__ = 'Copyright 2012-2016 Cédric Félizard'
-
-
-class Zeyple:
-    """Zeyple Encrypts Your Precious Log Emails"""
-
-    def __init__(self, config_fname='zeyple.conf'):
-        self.config = self.load_configuration(config_fname)
-
-        log_file = self.config.get('zeyple', 'log_file')
-        logging.basicConfig(
-            filename=log_file, level=logging.DEBUG,
-            format='%(asctime)s %(process)s %(levelname)s %(message)s'
-        )
-        logging.info("Zeyple ready to encrypt outgoing emails")
-
-    def load_configuration(self, filename):
-        """Reads and parses the config file"""
-
-        config = SafeConfigParser()
-        config.read([
-            os.path.join('/etc/', filename),
-            filename,
-        ])
-        if not config.sections():
-            raise IOError('Cannot open config file.')
-        return config
-
-    @property
-    def gpg(self):
-        protocol = gpgme.PROTOCOL_OpenPGP
-
-        if self.config.has_option('gpg', 'executable'):
-            executable = self.config.get('gpg', 'executable')
-        else:
-            executable = None  # Default value
-
-        home_dir = self.config.get('gpg', 'home')
-
-        ctx = gpgme.Context()
-        ctx.set_engine_info(protocol, executable, home_dir)
-        ctx.armor = True
-
-        return ctx
-
-    def process_message(self, message_data, recipients):
-        """Encrypts the message with recipient keys"""
-        message_data = encode_string(message_data)
-
-        in_message = message_from_binary(message_data)
-        logging.info(
-            "Processing outgoing message %s", in_message['Message-id'])
-
-        if not recipients:
-            logging.warn("Cannot find any recipients, ignoring")
-
-        sent_messages = []
-        for recipient in recipients:
-            logging.info("Recipient: %s", recipient)
-
-            key_id = self._user_key(recipient)
-            logging.info("Key ID: %s", key_id)
-
-            if key_id:
-                out_message = self._encrypt_message(in_message, key_id)
-
-                # Delete Content-Transfer-Encoding if present to default to
-                # "7bit" otherwise Thunderbird seems to hang in some cases.
-                del out_message["Content-Transfer-Encoding"]
-            else:
-                logging.warn("No keys found, message will be sent unencrypted")
-                out_message = copy.copy(in_message)
-
-            self._add_zeyple_header(out_message)
-            self._send_message(out_message, recipient)
-            sent_messages.append(out_message)
-
-        return sent_messages
-
-    def _get_version_part(self):
-        ret = email.mime.application.MIMEApplication(
-            'Version: 1\n',
-            'pgp-encrypted',
-            email.encoders.encode_noop,
-        )
-        ret.add_header(
-            'Content-Description',
-            "PGP/MIME version identification",
-        )
-        return ret
-
-    def _get_encrypted_part(self, payload):
-        ret = email.mime.application.MIMEApplication(
-            payload,
-            'octet-stream',
-            email.encoders.encode_noop,
-            name="encrypted.asc",
-        )
-        ret.add_header('Content-Description', "OpenPGP encrypted message")
-        ret.add_header(
-            'Content-Disposition',
-            'inline',
-            filename='encrypted.asc',
-        )
-        return ret
-
-    def _encrypt_message(self, in_message, key_id):
-        if in_message.is_multipart():
-            # get the body (after the first \n\n)
-            payload = in_message.as_string().split("\n\n", 1)[1].strip()
-
-            # prepend the Content-Type including the boundary
-            content_type = "Content-Type: " + in_message["Content-Type"]
-            payload = content_type + "\n\n" + payload
-
-            message = email.message.Message()
-            message.set_payload(payload)
-
-            payload = message.get_payload()
-
-        else:
-            payload = in_message.get_payload()
-            payload = encode_string(payload)
-
-            quoted_printable = email.charset.Charset('ascii')
-            quoted_printable.body_encoding = email.charset.QP
-
-            message = email.mime.nonmultipart.MIMENonMultipart(
-                'text', 'plain', charset='utf-8'
-            )
-            message.set_payload(payload, charset=quoted_printable)
-
-            mixed = email.mime.multipart.MIMEMultipart(
-                'mixed',
-                None,
-                [message],
-            )
-
-            # remove superfluous header
-            del mixed['MIME-Version']
-
-            payload = as_binary_string(mixed)
-
-        encrypted_payload = self._encrypt_payload(payload, [key_id])
-
-        version = self._get_version_part()
-        encrypted = self._get_encrypted_part(encrypted_payload)
-
-        out_message = copy.copy(in_message)
-        out_message.preamble = "This is an OpenPGP/MIME encrypted " \
-                               "message (RFC 4880 and 3156)"
-
-        if 'Content-Type' not in out_message:
-            out_message['Content-Type'] = 'multipart/encrypted'
-        else:
-            out_message.replace_header(
-                'Content-Type',
-                'multipart/encrypted',
-            )
-
-        out_message.set_param('protocol', 'application/pgp-encrypted')
-        out_message.set_payload([version, encrypted])
-
-        return out_message
-
-    def _encrypt_payload(self, payload, key_ids):
-        """Encrypts the payload with the given keys"""
-        payload = encode_string(payload)
-
-        plaintext = BytesIO(payload)
-        ciphertext = BytesIO()
-
-        self.gpg.armor = True
-
-        recipient = [self.gpg.get_key(key_id) for key_id in key_ids]
-
-        self.gpg.encrypt(recipient, gpgme.ENCRYPT_ALWAYS_TRUST,
-                         plaintext, ciphertext)
-
-        return ciphertext.getvalue()
-
-    def _user_key(self, email):
-        """Returns the GPG key for the given email address"""
-        logging.info("Trying to encrypt for %s", email)
-        keys = [key for key in self.gpg.keylist(email)]
-
-        if keys:
-            key = keys.pop()  # NOTE: looks like keys[0] is the master key
-            key_id = key.subkeys[0].keyid
-            return key_id
-
-        return None
-
-    def _add_zeyple_header(self, message):
-        if self.config.has_option('zeyple', 'add_header') and \
-           self.config.getboolean('zeyple', 'add_header'):
-            message.add_header(
-                'X-Zeyple',
-                "processed by {0} v{1}".format(__title__, __version__)
-            )
-
-    def _send_message(self, message, recipient):
-        """Sends the given message through the SMTP relay"""
-        logging.info("Sending message %s", message['Message-id'])
-
-        smtp = smtplib.SMTP(self.config.get('relay', 'host'),
-                            self.config.get('relay', 'port'))
-
-        smtp.sendmail(message['From'], recipient, message.as_string())
-        smtp.quit()
-
-        logging.info("Message %s sent", message['Message-id'])
-
-
-if __name__ == '__main__':
-    recipients = sys.argv[1:]
-
-    # BBB: Python 2.7 support
-    binary_stdin = sys.stdin.buffer if PY3K else sys.stdin
-    message = binary_stdin.read()
-
-    zeyple = Zeyple()
-    zeyple.process_message(message, recipients)

data/Dockerfiles/rspamd/Dockerfile

@@ -1,27 +1,29 @@
-FROM ubuntu:bionic
+FROM debian:buster-slim
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
 ARG DEBIAN_FRONTEND=noninteractive
+ARG CODENAME=buster
 ENV LC_ALL C
 
 RUN apt-get update && apt-get install -y \
   tzdata \
-	ca-certificates \
+  ca-certificates \
-	gnupg2 \
+  gnupg2 \
-	apt-transport-https \
+  apt-transport-https \
-	&& apt-key adv --fetch-keys https://rspamd.com/apt/gpg.key \
+  dnsutils \
-	&& echo "deb https://rspamd.com/apt-stable/ bionic main" > /etc/apt/sources.list.d/rspamd.list \
+  && apt-key adv --fetch-keys https://rspamd.com/apt-stable/gpg.key \
-	&& apt-get update && apt-get install -y rspamd \
+  && echo "deb [arch=amd64] https://rspamd.com/apt-stable/ $CODENAME main" > /etc/apt/sources.list.d/rspamd.list \
-	&& rm -rf /var/lib/apt/lists/* \
+  && echo "deb-src [arch=amd64] https://rspamd.com/apt-stable/ $CODENAME main" >> /etc/apt/sources.list.d/rspamd.list \
-	&& echo '.include $LOCAL_CONFDIR/local.d/rspamd.conf.local' > /etc/rspamd/rspamd.conf.local \
+  && apt-get update \
-	&& apt-get autoremove --purge \
+  && apt-get --no-install-recommends -y install rspamd \
-	&& apt-get clean \
+  && rm -rf /var/lib/apt/lists/* \
-	&& mkdir -p /run/rspamd \
+  && apt-get autoremove --purge \
-	&& chown _rspamd:_rspamd /run/rspamd
+  && apt-get clean \
+  && mkdir -p /run/rspamd \
+  && chown _rspamd:_rspamd /run/rspamd
 
 COPY settings.conf /etc/rspamd/settings.conf
 COPY docker-entrypoint.sh /docker-entrypoint.sh
-COPY metadata_exporter.lua /usr/share/rspamd/lua/metadata_exporter.lua
 
 ENTRYPOINT ["/docker-entrypoint.sh"]
 

data/Dockerfiles/rspamd/docker-entrypoint.sh

@@ -1,9 +1,37 @@
 #!/bin/bash
 
-chown -R _rspamd:_rspamd /var/lib/rspamd /etc/rspamd/local.d /etc/rspamd/override.d /etc/rspamd/custom
+mkdir -p /etc/rspamd/plugins.d \
+  /etc/rspamd/custom
+
+touch /etc/rspamd/rspamd.conf.local \
+  /etc/rspamd/rspamd.conf.override
+
 chmod 755 /var/lib/rspamd
-[[ ! -f /etc/rspamd/override.d/worker-controller-password.inc ]] && echo '# Placeholder' > /etc/rspamd/override.d/worker-controller-password.inc
+
-chown _rspamd:_rspamd /etc/rspamd/override.d/worker-controller-password.inc
+[[ ! -f /etc/rspamd/override.d/worker-controller-password.inc ]] && echo '# Autogenerated by mailcow' > /etc/rspamd/override.d/worker-controller-password.inc
-[[ ! -f /etc/rspamd/custom/sa-rules-heinlein ]] && echo '# to be auto-filled by dovecot-mailcow' > /etc/rspamd/custom/sa-rules-heinlein
+[[ ! -f /etc/rspamd/custom/sa-rules-heinlein ]] && echo '# Autogenerated by mailcow' > /etc/rspamd/custom/sa-rules-heinlein
+[[ ! -f /etc/rspamd/custom/dovecot_trusted.map ]] && echo '# Autogenerated by mailcow' > /etc/rspamd/custom/dovecot_trusted.map
+
+DOVECOT_V4=
+DOVECOT_V6=
+until [[ ! -z ${DOVECOT_V4} ]]; do
+  DOVECOT_V4=$(dig a dovecot +short)
+  DOVECOT_V6=$(dig aaaa dovecot +short)
+  [[ ! -z ${DOVECOT_V4} ]] && break;
+  echo "Waiting for Dovecot"
+  sleep 3
+done
+echo ${DOVECOT_V4}/32 > /etc/rspamd/custom/dovecot_trusted.map
+if [[ ! -z ${DOVECOT_V6} ]]; then
+  echo ${DOVECOT_V6}/128 >> /etc/rspamd/custom/dovecot_trusted.map
+fi
+
+chown -R _rspamd:_rspamd /var/lib/rspamd \
+  /etc/rspamd/local.d \
+  /etc/rspamd/override.d \
+  /etc/rspamd/custom \
+  /etc/rspamd/rspamd.conf.local \
+  /etc/rspamd/rspamd.conf.override \
+  /etc/rspamd/plugins.d
 
 exec "$@"

data/Dockerfiles/sogo/Dockerfile

@@ -3,7 +3,7 @@ LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 
 ARG DEBIAN_FRONTEND=noninteractive
 ENV LC_ALL C
-ENV GOSU_VERSION 1.9
+ENV GOSU_VERSION 1.11
 
 # Prerequisites
 RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -13,6 +13,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
   gettext \
   gnupg \
   mysql-client \
+  rsync \
   supervisor \
   syslog-ng \
   syslog-ng-core \
@@ -22,23 +23,19 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
   psmisc \
   wget \
   patch \
-  && rm -rf /var/lib/apt/lists/* \
   && dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
   && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
   && chmod +x /usr/local/bin/gosu \
-  && gosu nobody true
+  && gosu nobody true \
-
+  && mkdir /usr/share/doc/sogo \
-RUN mkdir /usr/share/doc/sogo \
   && touch /usr/share/doc/sogo/empty.sh \
   && apt-key adv --keyserver keyserver.ubuntu.com --recv-key 0x810273C4 \
   && echo "deb http://packages.inverse.ca/SOGo/nightly/4/debian/ stretch stretch" > /etc/apt/sources.list.d/sogo.list \
-  && apt-get update && apt-get install -y --force-yes \
+  && apt-get update && apt-get install -y --no-install-recommends \
     sogo \
     sogo-activesync \
+  && apt-get autoclean \
   && rm -rf /var/lib/apt/lists/* \
-  && echo '* * * * *   sogo   /usr/sbin/sogo-ealarms-notify -p /etc/sogo/sieve.creds 2>/dev/null' > /etc/cron.d/sogo \
-  && echo '* * * * *   sogo   /usr/sbin/sogo-tool expire-sessions 60' >> /etc/cron.d/sogo \
-  && echo '0 0 * * *   sogo   /usr/sbin/sogo-tool update-autoreply -p /etc/sogo/sieve.creds' >> /etc/cron.d/sogo \
   && touch /etc/default/locale
 
 COPY ./bootstrap-sogo.sh /bootstrap-sogo.sh
@@ -51,7 +48,3 @@ RUN chmod +x /bootstrap-sogo.sh \
   /usr/local/sbin/stop-supervisor.sh
 
 CMD exec /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
-
-VOLUME /usr/lib/GNUstep/SOGo/
-
-RUN rm -rf /tmp/* /var/tmp/*

data/Dockerfiles/sogo/bootstrap-sogo.sh

@@ -14,11 +14,11 @@ do
 done
 
 # Wait for updated schema
-DBV_NOW=$(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e "SELECT version FROM versions;" -BN)
+DBV_NOW=$(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e "SELECT version FROM versions WHERE application = 'db_schema';" -BN)
 DBV_NEW=$(grep -oE '\$db_version = .*;' init_db.inc.php | sed 's/$db_version = //g;s/;//g' | cut -d \" -f2)
 while [[ ${DBV_NOW} != ${DBV_NEW} ]]; do
   echo "Waiting for schema update..."
-  DBV_NOW=$(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e "SELECT version FROM versions;" -BN)
+  DBV_NOW=$(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e "SELECT version FROM versions WHERE application = 'db_schema';" -BN)
   DBV_NEW=$(grep -oE '\$db_version = .*;' init_db.inc.php | sed 's/$db_version = //g;s/;//g' | cut -d \" -f2)
   sleep 5
 done
@@ -30,10 +30,11 @@ mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -e
 
 while [[ ${VIEW_OK} != 'OK' ]]; do
   mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} << EOF
-CREATE VIEW sogo_view (c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, kind, multiple_bookings) AS
+CREATE VIEW sogo_view (c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, ext_acl, kind, multiple_bookings) AS
-SELECT mailbox.username, mailbox.domain, mailbox.username, if(json_extract(attributes, '$.force_pw_update') LIKE '%0%', if(json_extract(attributes, '$.sogo_access') LIKE '%1%', password, 'invalid'), 'invalid'), mailbox.name, mailbox.username, IFNULL(GROUP_CONCAT(ga.aliases SEPARATOR ' '), ''), IFNULL(gda.ad_alias, ''), mailbox.kind, mailbox.multiple_bookings FROM mailbox
+SELECT mailbox.username, mailbox.domain, mailbox.username, if(json_extract(attributes, '$.force_pw_update') LIKE '%0%', if(json_extract(attributes, '$.sogo_access') LIKE '%1%', password, '{SSHA256}A123A123A321A321A321B321B321B123B123B321B432F123E321123123321321'), '{SSHA256}A123A123A321A321A321B321B321B123B123B321B432F123E321123123321321'), mailbox.name, mailbox.username, IFNULL(GROUP_CONCAT(ga.aliases SEPARATOR ' '), ''), IFNULL(gda.ad_alias, ''), IFNULL(external_acl.send_as_acl, ''), mailbox.kind, mailbox.multiple_bookings FROM mailbox
 LEFT OUTER JOIN grouped_mail_aliases ga ON ga.username REGEXP CONCAT('(^|,)', mailbox.username, '($|,)')
 LEFT OUTER JOIN grouped_domain_alias_address gda ON gda.username = mailbox.username
+LEFT OUTER JOIN grouped_sender_acl_external external_acl ON external_acl.username = mailbox.username
 WHERE mailbox.active = '1'
 GROUP BY mailbox.username;
 EOF
@@ -51,7 +52,7 @@ while [[ ${STATIC_VIEW_OK} != 'OK' ]]; do
   if [[ ! -z $(mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -B -e "SELECT 'OK' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = '_sogo_static_view'") ]]; then
     STATIC_VIEW_OK=OK
     echo "Updating _sogo_static_view content..."
-    mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -B -e "REPLACE INTO _sogo_static_view (c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, kind, multiple_bookings) SELECT c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, kind, multiple_bookings from sogo_view;"
+    mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -B -e "REPLACE INTO _sogo_static_view (c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, ext_acl, kind, multiple_bookings) SELECT c_uid, domain, c_name, c_password, c_cn, mail, aliases, ad_aliases, ext_acl, kind, multiple_bookings from sogo_view;"
     mysql --socket=/var/run/mysqld/mysqld.sock -u ${DBUSER} -p${DBPASS} ${DBNAME} -B -e "DELETE FROM _sogo_static_view WHERE c_uid NOT IN (SELECT username FROM mailbox WHERE active = '1')"
   else
     echo "Waiting for database initialization..."
@@ -83,9 +84,16 @@ EOF
 done
 
 
-mkdir -p /var/lib/sogo/GNUstep/Defaults/
+if [[ "${ALLOW_ADMIN_EMAIL_LOGIN}" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
+  TRUST_PROXY="YES"
+else
+  TRUST_PROXY="NO"
+fi
+# cat /dev/urandom seems to hang here occasionally and is not recommended anyway, better use openssl
+RAND_PASS=$(openssl rand -base64 16 | tr -dc _A-Z-a-z-0-9)
 
 # Generate plist header with timezone data
+mkdir -p /var/lib/sogo/GNUstep/Defaults/
 cat <<EOF > /var/lib/sogo/GNUstep/Defaults/sogod.plist
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE plist PUBLIC "-//GNUstep//DTD plist 0.9//EN" "http://www.gnustep.org/plist-0_9.xml">
@@ -93,6 +101,12 @@ cat <<EOF > /var/lib/sogo/GNUstep/Defaults/sogod.plist
 <dict>
     <key>OCSAclURL</key>
     <string>mysql://${DBUSER}:${DBPASS}@%2Fvar%2Frun%2Fmysqld%2Fmysqld.sock/${DBNAME}/sogo_acl</string>
+    <key>SOGoIMAPServer</key>
+    <string>imaps://${IPV4_NETWORK}.250:993</string>
+    <key>SOGoTrustProxyAuthentication</key>
+    <string>${TRUST_PROXY}</string>
+    <key>SOGoEncryptionKey</key>
+    <string>${RAND_PASS}</string>
     <key>OCSCacheFolderURL</key>
     <string>mysql://${DBUSER}:${DBPASS}@%2Fvar%2Frun%2Fmysqld%2Fmysqld.sock/${DBNAME}/sogo_cache_folder</string>
     <key>OCSEMailAlarmsFolderURL</key>
@@ -125,6 +139,7 @@ while read -r line gal
                     <array>
                         <string>aliases</string>
                         <string>ad_aliases</string>
+                        <string>ext_acl</string>
                     </array>
                     <key>KindFieldName</key>
                     <string>kind</string>
@@ -168,19 +183,29 @@ chown sogo:sogo -R /var/lib/sogo/
 chmod 600 /var/lib/sogo/GNUstep/Defaults/sogod.plist
 
 # Patch ACLs
-if [[ ${ACL_ANYONE} == 'allow' ]]; then
+#if [[ ${ACL_ANYONE} == 'allow' ]]; then
-  #enable any or authenticated targets for ACL
+#  #enable any or authenticated targets for ACL
-  if patch -R -sfN --dry-run /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff > /dev/null; then
+#  if patch -R -sfN --dry-run /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff > /dev/null; then
-    patch -R /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff;
+#    patch -R /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff;
-  fi
+#  fi
-else
+#else
-  #disable any or authenticated targets for ACL
+#  #disable any or authenticated targets for ACL
-  if patch -sfN --dry-run /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff > /dev/null; then
+#  if patch -sfN --dry-run /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff > /dev/null; then
-    patch /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff;
+#    patch /usr/lib/GNUstep/SOGo/Templates/UIxAclEditor.wox < /acl.diff;
-  fi
+#  fi
-fi
+#fi
 
 # Copy logo, if any
 [[ -f /etc/sogo/sogo-full.svg ]] && cp /etc/sogo/sogo-full.svg /usr/lib/GNUstep/SOGo/WebServerResources/img/sogo-full.svg
 
+# Rsync web content
+echo "Syncing web content with named volume"
+rsync -a /usr/lib/GNUstep/SOGo/. /sogo_web/
+
+# Creating cronjobs
+echo "* * * * *   sogo   /usr/sbin/sogo-ealarms-notify -p /etc/sogo/sieve.creds 2>/dev/null" > /etc/cron.d/sogo
+echo "* * * * *   sogo   /usr/sbin/sogo-tool expire-sessions ${SOGO_EXPIRE_SESSION}" >> /etc/cron.d/sogo
+echo "0 0 * * *   sogo   /usr/sbin/sogo-tool update-autoreply -p /etc/sogo/sieve.creds" >> /etc/cron.d/sogo
+
+
 exec gosu sogo /usr/sbin/sogod

data/Dockerfiles/solr/Dockerfile

@@ -1,9 +1,25 @@
-FROM solr:7-alpine
+FROM solr:7.7-slim
-USER root
-COPY docker-entrypoint.sh /
 
-RUN apk --no-cache add su-exec curl tzdata \
+USER root
+
+ENV GOSU_VERSION 1.11
+
+COPY docker-entrypoint.sh /
+COPY solr-config-7.7.0.xml /
+COPY solr-schema-7.7.0.xml /
+
+RUN dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
+  && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
+  && chmod +x /usr/local/bin/gosu \
+  && gosu nobody true \
+  && apt-get update && apt-get install -y --no-install-recommends \
+  tzdata \
+  curl \
+  bash \
+  && apt-get autoclean \
+  && rm -rf /var/lib/apt/lists/* \
   && chmod +x /docker-entrypoint.sh \
-  && /docker-entrypoint.sh --bootstrap
+  && sync \
+  && bash /docker-entrypoint.sh --bootstrap
 
 ENTRYPOINT ["/docker-entrypoint.sh"]

data/Dockerfiles/solr/docker-entrypoint.sh

@@ -18,403 +18,44 @@ fi
 
 set -e
 
-# allow easier debugging with `docker run -e VERBOSE=yes`
-if [[ "$VERBOSE" = "yes" ]]; then
-  set -x
-fi
-
 # run the optional initdb
 . /opt/docker-solr/scripts/run-initdb
 
-function solr_config() {
-  curl -XPOST http://localhost:8983/solr/dovecot/schema -H 'Content-type:application/json' -d '{
-    "add-field-type":{
-      "name":"long",
-      "class":"solr.TrieLongField"
-    },
-    "add-field-type":{
-      "name":"dovecot_text",
-      "class":"solr.TextField",
-      "autoGeneratePhraseQueries":true,
-      "positionIncrementGap":100,
-      "indexAnalyser":{
-        "charFilter":{
-          "class":"solr.MappingCharFilterFactory",
-          "mapping":"mapping-FoldToASCII.txt"
-        },
-        "charFilter":{
-          "class":"solr.MappingCharFilterFactory",
-          "mapping":"mapping-ISOLatin1Accent.txt"
-        },
-        "charFilter":{
-          "class":"solr.HTMLStripCharFilterFactory"
-        },
-        "tokenizer":{
-          "class":"solr.StandardTokenizerFactory"
-        },
-        "filter":{
-          "class":"solr.StopFilterFactory",
-          "words":"stopwords.txt",
-          "ignoreCase":true
-        },
-        "filter":{
-          "class":"solr.WordDelimiterGraphFilterFactory",
-          "generateWordParts":1,
-          "generateNumberParts":1,
-          "splitOnCaseChange":1,
-          "splitOnNumerics":1,
-          "catenateWords":1,
-          "catenateNumbers":1,
-          "catenateAll":1
-        },
-        "filter":{
-          "class":"solr.FlattenGraphFilterFactory"
-        },
-        "filter":{
-          "class":"solr.LowerCaseFilterFactory"
-        },
-        "filter":{
-          "class":"solr.KeywordMarkerFilterFactory",
-          "protected":"protwords.txt"
-        },
-        "filter":{
-          "class":"solr.PorterStemFilterFactory"
-        }
-      },
-      "queryAnalyzer":{
-        "tokenizer":{
-          "class":"solr.StandardTokenizerFactory"
-        },
-        "filter":{
-          "class":"solr.SynonymGraphFilterFactory",
-          "expand":true,
-          "ignoreCase":true,
-          "synonyms":synonyms.txt
-        },
-        "filter":{
-          "class":"solr.FlattenGraphFilterFactory"
-        },
-        "filter":{
-          "class":"solr.StopFilterFactory",
-          "words":"stopwords.txt",
-          "ignoreCase":true
-        },
-        "filter":{
-          "class":"solr.WordDelimiterGraphFilterFactory",
-          "generateWordParts":1,
-          "generateNumberParts":1,
-          "splitOnCaseChange":1,
-          "splitOnNumerics":1,
-          "catenateWords":1,
-          "catenateNumbers":1,
-          "catenateAll":1
-        },
-        "filter":{
-          "class":"solr.LowerCaseFilterFactory"
-        },
-        "filter":{
-          "class":"solr.KeywordMarkerFilterFactory",
-          "protected":"protwords.txt"
-        },
-        "filter":{
-          "class":"solr.PorterStemFilterFactory"
-        }
-      }
-    },
-    "add-field":{
-      "name":"uid",
-      "type":"long",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "add-field":{
-      "name":"box",
-      "type":"string",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "add-field":{
-      "name":"user",
-      "type":"string",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "add-field":{
-      "name":"hdr",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-
-    },
-    "add-field":{
-      "name":"body",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "add-field":{
-      "name":"from",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "add-field":{
-      "name":"to",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "add-field":{
-      "name":"cc",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "add-field":{
-      "name":"bcc",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "add-field":{
-      "name":"subject",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    }
-  }'
-
-  curl -XPOST http://localhost:8983/solr/dovecot/schema -H 'Content-type:application/json' -d '{
-    "replace-field-type":{
-      "name":"long",
-      "class":"solr.TrieLongField"
-    },
-    "replace-field-type":{
-      "name":"dovecot_text",
-      "class":"solr.TextField",
-      "autoGeneratePhraseQueries":true,
-      "positionIncrementGap":100,
-      "indexAnalyser":{
-        "charFilter":{
-          "class":"solr.MappingCharFilterFactory",
-          "mapping":"mapping-FoldToASCII.txt"
-        },
-        "charFilter":{
-          "class":"solr.MappingCharFilterFactory",
-          "mapping":"mapping-ISOLatin1Accent.txt"
-        },
-        "charFilter":{
-          "class":"solr.HTMLStripCharFilterFactory"
-        },
-        "tokenizer":{
-          "class":"solr.StandardTokenizerFactory"
-        },
-        "filter":{
-          "class":"solr.StopFilterFactory",
-          "words":"stopwords.txt",
-          "ignoreCase":true
-        },
-        "filter":{
-          "class":"solr.WordDelimiterGraphFilterFactory",
-          "generateWordParts":1,
-          "generateNumberParts":1,
-          "splitOnCaseChange":1,
-          "splitOnNumerics":1,
-          "catenateWords":1,
-          "catenateNumbers":1,
-          "catenateAll":1
-        },
-        "filter":{
-          "class":"solr.FlattenGraphFilterFactory"
-        },
-        "filter":{
-          "class":"solr.LowerCaseFilterFactory"
-        },
-        "filter":{
-          "class":"solr.KeywordMarkerFilterFactory",
-          "protected":"protwords.txt"
-        },
-        "filter":{
-          "class":"solr.PorterStemFilterFactory"
-        }
-      },
-      "queryAnalyzer":{
-        "tokenizer":{
-          "class":"solr.StandardTokenizerFactory"
-        },
-        "filter":{
-          "class":"solr.SynonymGraphFilterFactory",
-          "expand":true,
-          "ignoreCase":true,
-          "synonyms":synonyms.txt
-        },
-        "filter":{
-          "class":"solr.FlattenGraphFilterFactory"
-        },
-        "filter":{
-          "class":"solr.StopFilterFactory",
-          "words":"stopwords.txt",
-          "ignoreCase":true
-        },
-        "filter":{
-          "class":"solr.WordDelimiterGraphFilterFactory",
-          "generateWordParts":1,
-          "generateNumberParts":1,
-          "splitOnCaseChange":1,
-          "splitOnNumerics":1,
-          "catenateWords":1,
-          "catenateNumbers":1,
-          "catenateAll":1
-        },
-        "filter":{
-          "class":"solr.LowerCaseFilterFactory"
-        },
-        "filter":{
-          "class":"solr.KeywordMarkerFilterFactory",
-          "protected":"protwords.txt"
-        },
-        "filter":{
-          "class":"solr.PorterStemFilterFactory"
-        }
-      }
-    },
-    "replace-field":{
-      "name":"uid",
-      "type":"long",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "replace-field":{
-      "name":"box",
-      "type":"string",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "replace-field":{
-      "name":"user",
-      "type":"string",
-      "indexed":true,
-      "stored":true,
-      "required":true
-    },
-    "replace-field":{
-      "name":"hdr",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-
-    },
-    "replace-field":{
-      "name":"body",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "replace-field":{
-      "name":"from",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "replace-field":{
-      "name":"to",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "replace-field":{
-      "name":"cc",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "replace-field":{
-      "name":"bcc",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    },
-    "replace-field":{
-      "name":"subject",
-      "type":"dovecot_text",
-      "indexed":true,
-      "stored":false
-    }
-  }'
-
-  curl -XPOST http://localhost:8983/solr/dovecot/config -H 'Content-type:application/json' -d '{
-    "update-requesthandler":{
-      "name":"/select",
-      "class":"solr.SearchHandler",
-      "defaults":{
-        "wt":"xml"
-      }
-    }
-  }'
-
-  curl -XPOST http://localhost:8983/solr/dovecot/config/updateHandler -d '{
-    "set-property": {
-      "updateHandler.autoSoftCommit.maxDocs":500,
-      "updateHandler.autoSoftCommit.maxTime":120000,
-      "updateHandler.autoCommit.maxDocs":200,
-      "updateHandler.autoCommit.maxTime":1800000,
-      "updateHandler.autoCommit.openSearcher":false
-    }
-  }'
-}
-
 # fixing volume permission
-
+[[ -d /opt/solr/server/solr/dovecot-fts/data ]] && chown -R solr:solr /opt/solr/server/solr/dovecot-fts/data
-[[ -d /opt/solr/server/solr/dovecot/data ]] && chown -R solr:solr /opt/solr/server/solr/dovecot/data
 if [[ "${1}" != "--bootstrap" ]]; then
   sed -i '/SOLR_HEAP=/c\SOLR_HEAP="'${SOLR_HEAP:-1024}'m"' /opt/solr/bin/solr.in.sh
 else
   sed -i '/SOLR_HEAP=/c\SOLR_HEAP="256m"' /opt/solr/bin/solr.in.sh
 fi
 
-# start a Solr so we can use the Schema API, but only on localhost,
+if [[ "${1}" == "--bootstrap" ]]; then
-# so that clients don't see Solr until we have configured it.
+  echo "Creating initial configuration"
+  echo "Modifying default config set"
+  cp /solr-config-7.7.0.xml /opt/solr/server/solr/configsets/_default/conf/solrconfig.xml
+  cp /solr-schema-7.7.0.xml /opt/solr/server/solr/configsets/_default/conf/schema.xml
+  rm /opt/solr/server/solr/configsets/_default/conf/managed-schema
 
-echo "Starting local Solr instance to setup configuration"
+  echo "Starting local Solr instance to setup configuration"
-su-exec solr start-local-solr
+  gosu solr start-local-solr
 
-# keep a sentinel file so we don't try to create the core a second time
+  echo "Creating core \"dovecot-fts\""
-# for example when we restart a container.
+  gosu solr /opt/solr/bin/solr create -c "dovecot-fts"
-
-SENTINEL=/opt/docker-solr/core_created
-
-if [[ -f ${SENTINEL} ]]; then
-  echo "skipping core creation"
-else
-  echo "Creating core \"dovecot\""
-  su-exec solr /opt/solr/bin/solr create -c "dovecot"
 
   # See https://github.com/docker-solr/docker-solr/issues/27
   echo "Checking core"
   while ! wget -O - 'http://localhost:8983/solr/admin/cores?action=STATUS' | grep -q instanceDir; do
     echo "Could not find any cores, waiting..."
-    sleep 5
+    sleep 3
   done
-  echo "Created core \"dovecot\""
-  touch ${SENTINEL}
-fi
 
-echo "Starting configuration"
+  echo "Created core \"dovecot-fts\""
-while ! wget -O - 'http://localhost:8983/solr/admin/cores?action=STATUS' | grep -q instanceDir; do
+
-  echo "Waiting for Solr..."
+  echo "Stopping local Solr"
-  sleep 5
+  gosu solr stop-local-solr
-done
-solr_config
-echo "Stopping local Solr"
-su-exec solr stop-local-solr
 
-if [[ "${1}" == "--bootstrap" ]]; then
   exit 0
-else
-  exec su-exec solr solr-foreground
 fi
+
+exec gosu solr solr-foreground
+

data/Dockerfiles/solr/solr-config-7.7.0.xml

@@ -0,0 +1,289 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+
+<!-- This is the default config with stuff non-essential to Dovecot removed. -->
+
+<config>
+  <!-- Controls what version of Lucene various components of Solr
+       adhere to.  Generally, you want to use the latest version to
+       get all bug fixes and improvements. It is highly recommended
+       that you fully re-index after changing this setting as it can
+       affect both how text is indexed and queried.
+  -->
+  <luceneMatchVersion>7.7.0</luceneMatchVersion>
+
+  <!-- A 'dir' option by itself adds any files found in the directory
+       to the classpath, this is useful for including all jars in a
+       directory.
+
+       When a 'regex' is specified in addition to a 'dir', only the
+       files in that directory which completely match the regex
+       (anchored on both ends) will be included.
+
+       If a 'dir' option (with or without a regex) is used and nothing
+       is found that matches, a warning will be logged.
+
+       The examples below can be used to load some solr-contribs along
+       with their external dependencies.
+    -->
+  <lib dir="${solr.install.dir:../../../..}/contrib/extraction/lib" regex=".*\.jar" />
+  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-cell-\d.*\.jar" />
+
+  <lib dir="${solr.install.dir:../../../..}/contrib/clustering/lib/" regex=".*\.jar" />
+  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-clustering-\d.*\.jar" />
+
+  <lib dir="${solr.install.dir:../../../..}/contrib/langid/lib/" regex=".*\.jar" />
+  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-langid-\d.*\.jar" />
+
+  <lib dir="${solr.install.dir:../../../..}/contrib/velocity/lib" regex=".*\.jar" />
+  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-velocity-\d.*\.jar" />
+
+  <!-- Data Directory
+
+       Used to specify an alternate directory to hold all index data
+       other than the default ./data under the Solr home.  If
+       replication is in use, this should match the replication
+       configuration.
+    -->
+  <dataDir>${solr.data.dir:}</dataDir>
+
+  <!-- The default high-performance update handler -->
+  <updateHandler class="solr.DirectUpdateHandler2">
+
+    <!-- Enables a transaction log, used for real-time get, durability, and
+         and solr cloud replica recovery.  The log can grow as big as
+         uncommitted changes to the index, so use of a hard autoCommit
+         is recommended (see below).
+         "dir" - the target directory for transaction logs, defaults to the
+                solr data directory.
+         "numVersionBuckets" - sets the number of buckets used to keep
+                track of max version values when checking for re-ordered
+                updates; increase this value to reduce the cost of
+                synchronizing access to version buckets during high-volume
+                indexing, this requires 8 bytes (long) * numVersionBuckets
+                of heap space per Solr core.
+    -->
+    <updateLog>
+      <str name="dir">${solr.ulog.dir:}</str>
+      <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
+    </updateLog>
+
+    <!-- AutoCommit
+
+         Perform a hard commit automatically under certain conditions.
+         Instead of enabling autoCommit, consider using "commitWithin"
+         when adding documents.
+
+         http://wiki.apache.org/solr/UpdateXmlMessages
+
+         maxDocs - Maximum number of documents to add since the last
+                   commit before automatically triggering a new commit.
+
+         maxTime - Maximum amount of time in ms that is allowed to pass
+                   since a document was added before automatically
+                   triggering a new commit.
+         openSearcher - if false, the commit causes recent index changes
+           to be flushed to stable storage, but does not cause a new
+           searcher to be opened to make those changes visible.
+
+         If the updateLog is enabled, then it's highly recommended to
+         have some sort of hard autoCommit to limit the log size.
+      -->
+    <autoCommit>
+      <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+      <openSearcher>false</openSearcher>
+    </autoCommit>
+
+    <!-- softAutoCommit is like autoCommit except it causes a
+         'soft' commit which only ensures that changes are visible
+         but does not ensure that data is synced to disk.  This is
+         faster and more near-realtime friendly than a hard commit.
+      -->
+    <autoSoftCommit>
+      <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
+    </autoSoftCommit>
+
+    <!-- Update Related Event Listeners
+
+         Various IndexWriter related events can trigger Listeners to
+         take actions.
+
+         postCommit - fired after every commit or optimize command
+         postOptimize - fired after every optimize command
+      -->
+
+  </updateHandler>
+
+  <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+       Query section - these settings control query time things like caches
+       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
+  <query>
+    <!-- Solr Internal Query Caches
+
+         There are two implementations of cache available for Solr,
+         LRUCache, based on a synchronized LinkedHashMap, and
+         FastLRUCache, based on a ConcurrentHashMap.
+
+         FastLRUCache has faster gets and slower puts in single
+         threaded operation and thus is generally faster than LRUCache
+         when the hit ratio of the cache is high (> 75%), and may be
+         faster under other scenarios on multi-cpu systems.
+    -->
+
+    <!-- Filter Cache
+
+         Cache used by SolrIndexSearcher for filters (DocSets),
+         unordered sets of *all* documents that match a query.  When a
+         new searcher is opened, its caches may be prepopulated or
+         "autowarmed" using data from caches in the old searcher.
+         autowarmCount is the number of items to prepopulate.  For
+         LRUCache, the autowarmed items will be the most recently
+         accessed items.
+
+         Parameters:
+           class - the SolrCache implementation LRUCache or
+               (LRUCache or FastLRUCache)
+           size - the maximum number of entries in the cache
+           initialSize - the initial capacity (number of entries) of
+               the cache.  (see java.util.HashMap)
+           autowarmCount - the number of entries to prepopulate from
+               and old cache.
+           maxRamMB - the maximum amount of RAM (in MB) that this cache is allowed
+                      to occupy. Note that when this option is specified, the size
+                      and initialSize parameters are ignored.
+      -->
+    <filterCache class="solr.FastLRUCache"
+                 size="512"
+                 initialSize="512"
+                 autowarmCount="0"/>
+
+    <!-- Query Result Cache
+
+         Caches results of searches - ordered lists of document ids
+         (DocList) based on a query, a sort, and the range of documents requested.
+         Additional supported parameter by LRUCache:
+            maxRamMB - the maximum amount of RAM (in MB) that this cache is allowed
+                       to occupy
+      -->
+    <queryResultCache class="solr.LRUCache"
+                      size="512"
+                      initialSize="512"
+                      autowarmCount="0"/>
+
+    <!-- Document Cache
+
+         Caches Lucene Document objects (the stored fields for each
+         document).  Since Lucene internal document ids are transient,
+         this cache will not be autowarmed.
+      -->
+    <documentCache class="solr.LRUCache"
+                   size="512"
+                   initialSize="512"
+                   autowarmCount="0"/>
+
+    <!-- custom cache currently used by block join -->
+    <cache name="perSegFilter"
+           class="solr.search.LRUCache"
+           size="10"
+           initialSize="0"
+           autowarmCount="10"
+           regenerator="solr.NoOpRegenerator" />
+
+    <!-- Lazy Field Loading
+
+         If true, stored fields that are not requested will be loaded
+         lazily.  This can result in a significant speed improvement
+         if the usual case is to not load all stored fields,
+         especially if the skipped fields are large compressed text
+         fields.
+    -->
+    <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+    <!-- Result Window Size
+
+         An optimization for use with the queryResultCache.  When a search
+         is requested, a superset of the requested number of document ids
+         are collected.  For example, if a search for a particular query
+         requests matching documents 10 through 19, and queryWindowSize is 50,
+         then documents 0 through 49 will be collected and cached.  Any further
+         requests in that range can be satisfied via the cache.
+      -->
+    <queryResultWindowSize>20</queryResultWindowSize>
+
+    <!-- Maximum number of documents to cache for any entry in the
+         queryResultCache.
+      -->
+    <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
+
+    <!-- Use Cold Searcher
+
+         If a search request comes in and there is no current
+         registered searcher, then immediately register the still
+         warming searcher and use it.  If "false" then all requests
+         will block until the first searcher is done warming.
+      -->
+    <useColdSearcher>false</useColdSearcher>
+
+  </query>
+
+
+  <!-- Request Dispatcher
+
+       This section contains instructions for how the SolrDispatchFilter
+       should behave when processing requests for this SolrCore.
+
+    -->
+  <requestDispatcher>
+    <httpCaching never304="true" />
+  </requestDispatcher>
+
+  <!-- Request Handlers
+
+       http://wiki.apache.org/solr/SolrRequestHandler
+
+       Incoming queries will be dispatched to a specific handler by name
+       based on the path specified in the request.
+
+       If a Request Handler is declared with startup="lazy", then it will
+       not be initialized until the first request that uses it.
+
+    -->
+  <!-- SearchHandler
+
+       http://wiki.apache.org/solr/SearchHandler
+
+       For processing Search Queries, the primary Request Handler
+       provided with Solr is "SearchHandler" It delegates to a sequent
+       of SearchComponents (see below) and supports distributed
+       queries across multiple shards
+    -->
+  <requestHandler name="/select" class="solr.SearchHandler">
+    <!-- default values for query parameters can be specified, these
+         will be overridden by parameters in the request
+      -->
+    <lst name="defaults">
+      <str name="echoParams">explicit</str>
+      <int name="rows">10</int>
+    </lst>
+  </requestHandler>
+
+  <initParams path="/update/**,/select">
+    <lst name="defaults">
+      <str name="df">_text_</str>
+    </lst>
+  </initParams>
+
+  <!-- Response Writers
+
+       http://wiki.apache.org/solr/QueryResponseWriter
+
+       Request responses will be written using the writer specified by
+       the 'wt' request parameter matching the name of a registered
+       writer.
+
+       The "default" writer is the default and will be used if 'wt' is
+       not specified in the request.
+    -->
+  <queryResponseWriter name="xml"
+                       default="true"
+                       class="solr.XMLResponseWriter" />
+</config>

data/Dockerfiles/solr/solr-schema-7.7.0.xml

@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<schema name="dovecot-fts" version="2.0">
+  <fieldType name="string" class="solr.StrField" omitNorms="true" sortMissingLast="true"/>
+  <fieldType name="long" class="solr.LongPointField" positionIncrementGap="0"/>
+  <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
+
+  <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100">
+    <analyzer type="index">
+      <tokenizer class="solr.StandardTokenizerFactory"/>
+      <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="20"/>
+      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
+      <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
+      <filter class="solr.FlattenGraphFilterFactory"/>
+      <filter class="solr.LowerCaseFilterFactory"/>
+      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
+      <filter class="solr.PorterStemFilterFactory"/>
+    </analyzer>
+    <analyzer type="query">
+      <tokenizer class="solr.StandardTokenizerFactory"/>
+      <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
+      <filter class="solr.FlattenGraphFilterFactory"/>
+      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
+      <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
+      <filter class="solr.LowerCaseFilterFactory"/>
+      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
+      <filter class="solr.PorterStemFilterFactory"/>
+    </analyzer>
+  </fieldType>
+
+  <field name="id" type="string" indexed="true" required="true" stored="true"/>
+  <field name="uid" type="long" indexed="true" required="true" stored="true"/>
+  <field name="box" type="string" indexed="true" required="true" stored="true"/>
+  <field name="user" type="string" indexed="true" required="true" stored="true"/>
+
+  <field name="hdr" type="text" indexed="true" stored="false"/>
+  <field name="body" type="text" indexed="true" stored="false"/>
+
+  <field name="from" type="text" indexed="true" stored="false"/>
+  <field name="to" type="text" indexed="true" stored="false"/>
+  <field name="cc" type="text" indexed="true" stored="false"/>
+  <field name="bcc" type="text" indexed="true" stored="false"/>
+  <field name="subject" type="text" indexed="true" stored="false"/>
+
+  <!-- Used by Solr internally: -->
+  <field name="_version_" type="long" indexed="true" stored="true"/>
+
+  <uniqueKey>id</uniqueKey>
+</schema>

data/Dockerfiles/unbound/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.9
+FROM alpine:3.10
 
 LABEL maintainer "Andre Peters <andre.peters@servercow.de>"
 

data/Dockerfiles/watchdog/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.9
+FROM alpine:3.10
 LABEL maintainer "André Peters <andre.peters@servercow.de>"
 
 # Installation
@@ -7,11 +7,13 @@ RUN apk add --update \
   nagios-plugins-tcp \
   nagios-plugins-http \
   nagios-plugins-ping \
+  mariadb-client \
   curl \
   bash \
   coreutils \
   jq \
   fcgi \
+  openssl \
   nagios-plugins-mysql \
   nagios-plugins-dns \
   nagios-plugins-disk \
@@ -26,11 +28,13 @@ RUN apk add --update \
   perl-term-readkey \
   tini \
   tzdata \
+  whois \
   && curl https://raw.githubusercontent.com/mludvig/smtp-cli/v3.9/smtp-cli -o /smtp-cli \
   && chmod +x smtp-cli
 
 COPY watchdog.sh /watchdog.sh
 
-ENTRYPOINT ["/sbin/tini", "-g", "--"]
+#ENTRYPOINT ["/sbin/tini", "-g", "--"]
 # Less verbose
+
 CMD /watchdog.sh 2> /dev/null

data/Dockerfiles/watchdog/watchdog.sh

@@ -5,6 +5,8 @@ trap "kill 0" EXIT
 
 # Prepare
 BACKGROUND_TASKS=()
+echo "Waiting for containers to settle..."
+sleep 10
 
 if [[ "${USE_WATCHDOG}" =~ ^([nN][oO]|[nN])+$ ]]; then
   echo -e "$(date) - USE_WATCHDOG=n, skipping watchdog..."
@@ -17,7 +19,28 @@ if [[ ! -p /tmp/com_pipe ]]; then
   mkfifo /tmp/com_pipe
 fi
 
+# Wait for containers
+while ! mysqladmin status --socket=/var/run/mysqld/mysqld.sock -u${DBUSER} -p${DBPASS} --silent; do
+  echo "Waiting for SQL..."
+  sleep 2
+done
+
+until [[ $(redis-cli -h redis-mailcow PING) == "PONG" ]]; do
+  echo "Waiting for Redis..."
+  sleep 2
+done
+
+redis-cli -h redis-mailcow DEL F2B_RES > /dev/null
+
 # Common functions
+array_diff() {
+  # https://stackoverflow.com/questions/2312762, Alex Offshore
+  eval local ARR1=\(\"\${$2[@]}\"\)
+  eval local ARR2=\(\"\${$3[@]}\"\)
+  local IFS=$'\n'
+  mapfile -t $1 < <(comm -23 <(echo "${ARR1[*]}" | sort) <(echo "${ARR2[*]}" | sort))
+}
+
 progress() {
   SERVICE=${1}
   TOTAL=${2}
@@ -37,7 +60,7 @@ progress() {
 log_msg() {
   if [[ ${2} != "no_redis" ]]; then
     redis-cli -h redis LPUSH WATCHDOG_LOG "{\"time\":\"$(date +%s)\",\"message\":\"$(printf '%s' "${1}" | \
-      tr '%&;$"_[]{}-\r\n' ' ')\"}" > /dev/null
+      tr '\r\n%&;$"_[]{}-' ' ')\"}" > /dev/null
   fi
   echo $(date) $(printf '%s\n' "${1}")
 }
@@ -46,6 +69,13 @@ function mail_error() {
   [[ -z ${1} ]] && return 1
   [[ -z ${2} ]] && BODY="Service was restarted on $(date), please check your mailcow installation." || BODY="$(date) - ${2}"
   WATCHDOG_NOTIFY_EMAIL=$(echo "${WATCHDOG_NOTIFY_EMAIL}" | sed 's/"//;s|"$||')
+  # Some exceptions for subject and body formats
+  if [[ ${1} == "fail2ban" ]]; then
+    SUBJECT="${BODY}"
+    BODY="Please see netfilter-mailcow for more details and triggered rules."
+  else
+    SUBJECT="Watchdog ALERT: ${1}"
+  fi
   IFS=',' read -r -a MAIL_RCPTS <<< "${WATCHDOG_NOTIFY_EMAIL}"
   for rcpt in "${MAIL_RCPTS[@]}"; do
     RCPT_DOMAIN=
@@ -56,15 +86,15 @@ function mail_error() {
       log_msg "Cannot determine MX for ${rcpt}, skipping email notification..."
       return 1
     fi
-    [ -f "/tmp/${1}" ] && ATTACH="--attach /tmp/${1}@text/plain" || ATTACH=
+    [ -f "/tmp/${1}" ] && BODY="/tmp/${1}"
-    ./smtp-cli --missing-modules-ok \
+    timeout 10s ./smtp-cli --missing-modules-ok \
-      --subject="Watchdog: ${1} hit the error rate limit" \
+      --charset=UTF-8 \
+      --subject="${SUBJECT}" \
       --body-plain="${BODY}" \
       --to=${rcpt} \
       --from="watchdog@${MAILCOW_HOSTNAME}" \
       --server="${RCPT_MX}" \
-      --hello-host=${MAILCOW_HOSTNAME} \
+      --hello-host=${MAILCOW_HOSTNAME}
-      ${ATTACH}
     log_msg "Sent notification email to ${rcpt}"
   done
 }
@@ -111,11 +141,11 @@ get_container_ip() {
 nginx_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=16
+  THRESHOLD=5
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/nginx-mailcow
+    touch /tmp/nginx-mailcow; echo "$(tail -50 /tmp/nginx-mailcow)" > /tmp/nginx-mailcow
     host_ip=$(get_container_ip nginx-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_http -4 -H ${host_ip} -u / -p 8081 2>> /tmp/nginx-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -127,7 +157,7 @@ nginx_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -136,11 +166,11 @@ nginx_checks() {
 unbound_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=8
+  THRESHOLD=5
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/unbound-mailcow
+    touch /tmp/unbound-mailcow; echo "$(tail -50 /tmp/unbound-mailcow)" > /tmp/unbound-mailcow
     host_ip=$(get_container_ip unbound-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_dns -s ${host_ip} -H stackoverflow.com 2>> /tmp/unbound-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -159,7 +189,32 @@ unbound_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
+    fi
+  done
+  return 1
+}
+
+redis_checks() {
+  err_count=0
+  diff_c=0
+  THRESHOLD=5
+  # Reduce error count by 2 after restarting an unhealthy container
+  trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
+  while [ ${err_count} -lt ${THRESHOLD} ]; do
+    touch /tmp/redis-mailcow; echo "$(tail -50 /tmp/redis-mailcow)" > /tmp/redis-mailcow
+    host_ip=$(get_container_ip redis-mailcow)
+    err_c_cur=${err_count}
+    /usr/lib/nagios/plugins/check_tcp -4 -H redis-mailcow -p 6379 -E -s "PING\n" -q "QUIT" -e "PONG" 2>> /tmp/redis-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
+    [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
+    [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
+    progress "Redis" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
+    if [[ $? == 10 ]]; then
+      diff_c=0
+      sleep 1
+    else
+      diff_c=0
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -168,11 +223,11 @@ unbound_checks() {
 mysql_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=12
+  THRESHOLD=5
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/mysql-mailcow
+    touch /tmp/mysql-mailcow; echo "$(tail -50 /tmp/mysql-mailcow)" > /tmp/mysql-mailcow
     host_ip=$(get_container_ip mysql-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_mysql -s /var/run/mysqld/mysqld.sock -u ${DBUSER} -p ${DBPASS} -d ${DBNAME} 2>> /tmp/mysql-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -185,7 +240,7 @@ mysql_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -194,11 +249,11 @@ mysql_checks() {
 sogo_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=10
+  THRESHOLD=5
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/sogo-mailcow
+    touch /tmp/sogo-mailcow; echo "$(tail -50 /tmp/sogo-mailcow)" > /tmp/sogo-mailcow
     host_ip=$(get_container_ip sogo-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_http -4 -H ${host_ip} -u /SOGo.index/ -p 20000 -R "SOGo\.MainUI" 2>> /tmp/sogo-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -210,7 +265,7 @@ sogo_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -223,10 +278,10 @@ postfix_checks() {
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/postfix-mailcow
+    touch /tmp/postfix-mailcow; echo "$(tail -50 /tmp/postfix-mailcow)" > /tmp/postfix-mailcow
     host_ip=$(get_container_ip postfix-mailcow)
     err_c_cur=${err_count}
-    /usr/lib/nagios/plugins/check_smtp -4 -H ${host_ip} -p 589 -f "watchdog@invalid" -C "RCPT TO:null@localhost" -C DATA -C . -R 250 2>> /tmp/postfix-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
+    /usr/lib/nagios/plugins/check_smtp -4 -H ${host_ip} -p 589 -f "watchdog@invalid" -C "RCPT TO:watchdog@localhost" -C DATA -C . -R 250 2>> /tmp/postfix-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
     /usr/lib/nagios/plugins/check_smtp -4 -H ${host_ip} -p 589 -S 2>> /tmp/postfix-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
     [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
     [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
@@ -236,7 +291,7 @@ postfix_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -245,11 +300,11 @@ postfix_checks() {
 clamd_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=5
+  THRESHOLD=15
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/clamd-mailcow
+    touch /tmp/clamd-mailcow; echo "$(tail -50 /tmp/clamd-mailcow)" > /tmp/clamd-mailcow
     host_ip=$(get_container_ip clamd-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_clamd -4 -H ${host_ip} 2>> /tmp/clamd-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -261,7 +316,7 @@ clamd_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 120 ) + 20 ))
     fi
   done
   return 1
@@ -270,11 +325,11 @@ clamd_checks() {
 dovecot_checks() {
   err_count=0
   diff_c=0
-  THRESHOLD=20
+  THRESHOLD=12
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/dovecot-mailcow
+    touch /tmp/dovecot-mailcow; echo "$(tail -50 /tmp/dovecot-mailcow)" > /tmp/dovecot-mailcow
     host_ip=$(get_container_ip dovecot-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_smtp -4 -H ${host_ip} -p 24 -f "watchdog@invalid" -C "RCPT TO:<watchdog@invalid>" -L -R "User doesn't exist" 2>> /tmp/dovecot-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -290,7 +345,7 @@ dovecot_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -303,7 +358,7 @@ phpfpm_checks() {
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/php-fpm-mailcow
+    touch /tmp/php-fpm-mailcow; echo "$(tail -50 /tmp/php-fpm-mailcow)" > /tmp/php-fpm-mailcow
     host_ip=$(get_container_ip php-fpm-mailcow)
     err_c_cur=${err_count}
     /usr/lib/nagios/plugins/check_tcp -H ${host_ip} -p 9001 2>> /tmp/php-fpm-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
@@ -316,7 +371,7 @@ phpfpm_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -344,7 +399,73 @@ ratelimit_checks() {
       sleep 1
     else
       diff_c=0
-      sleep $(( ( RANDOM % 30 )  + 10 ))
+      sleep $(( ( RANDOM % 60 ) + 20 ))
+    fi
+  done
+  return 1
+}
+
+fail2ban_checks() {
+  err_count=0
+  diff_c=0
+  THRESHOLD=1
+  F2B_LOG_STATUS=($(redis-cli -h redis-mailcow --raw HKEYS F2B_ACTIVE_BANS))
+  F2B_RES=
+  # Reduce error count by 2 after restarting an unhealthy container
+  trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
+  while [ ${err_count} -lt ${THRESHOLD} ]; do
+    err_c_cur=${err_count}
+    F2B_LOG_STATUS_PREV=(${F2B_LOG_STATUS[@]})
+    F2B_LOG_STATUS=($(redis-cli -h redis-mailcow --raw HKEYS F2B_ACTIVE_BANS))
+    array_diff F2B_RES F2B_LOG_STATUS F2B_LOG_STATUS_PREV
+    if [[ ! -z "${F2B_RES}" ]]; then
+      err_count=$(( ${err_count} + 1 ))
+      echo -n "${F2B_RES[@]}" | tr -cd "[a-fA-F0-9.:/] " | timeout 3s redis-cli -x -h redis-mailcow SET F2B_RES > /dev/null
+      if [ $? -ne 0 ]; then
+         redis-cli -x -h redis-mailcow DEL F2B_RES
+      fi
+    fi
+    [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
+    [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
+    progress "Fail2ban" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
+    if [[ $? == 10 ]]; then
+      diff_c=0
+      sleep 1
+    else
+      diff_c=0
+      sleep $(( ( RANDOM % 60 ) + 20 ))
+    fi
+  done
+  return 1
+}
+
+acme_checks() {
+  err_count=0
+  diff_c=0
+  THRESHOLD=1
+  ACME_LOG_STATUS=$(redis-cli -h redis GET ACME_FAIL_TIME)
+  if [[ -z "${ACME_LOG_STATUS}" ]]; then
+    redis-cli -h redis SET ACME_FAIL_TIME 0
+    ACME_LOG_STATUS=0
+  fi
+  # Reduce error count by 2 after restarting an unhealthy container
+  trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
+  while [ ${err_count} -lt ${THRESHOLD} ]; do
+    err_c_cur=${err_count}
+    ACME_LOG_STATUS_PREV=${ACME_LOG_STATUS}
+    ACME_LOG_STATUS=$(redis-cli -h redis GET ACME_FAIL_TIME)
+    if [[ ${ACME_LOG_STATUS_PREV} != ${ACME_LOG_STATUS} ]]; then
+      err_count=$(( ${err_count} + 1 ))
+    fi
+    [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
+    [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
+    progress "ACME" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
+    if [[ $? == 10 ]]; then
+      diff_c=0
+      sleep 1
+    else
+      diff_c=0
+      sleep $(( ( RANDOM % 60 ) + 20 ))
     fi
   done
   return 1
@@ -358,10 +479,11 @@ ipv6nat_checks() {
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
     err_c_cur=${err_count}
-    IPV6NAT_CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\")) | .id")
+    CONTAINERS=$(curl --silent --insecure https://dockerapi/containers/json)
+    IPV6NAT_CONTAINER_ID=$(echo ${CONTAINERS} | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\")) | .id")
     if [[ ! -z ${IPV6NAT_CONTAINER_ID} ]]; then
-      LATEST_STARTED="$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], StartedAt: .State.StartedAt}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\") | not)" | jq -rc .StartedAt | xargs -n1 date +%s -d | sort | tail -n1)"
+      LATEST_STARTED="$(echo ${CONTAINERS} | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], StartedAt: .State.StartedAt}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\") | not)" | jq -rc .StartedAt | xargs -n1 date +%s -d | sort | tail -n1)"
-      LATEST_IPV6NAT="$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], StartedAt: .State.StartedAt}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\"))" | jq -rc .StartedAt | xargs -n1 date +%s -d | sort | tail -n1)"
+      LATEST_IPV6NAT="$(echo ${CONTAINERS} | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], StartedAt: .State.StartedAt}" | jq -rc "select( .name | tostring | contains(\"ipv6nat-mailcow\"))" | jq -rc .StartedAt | xargs -n1 date +%s -d | sort | tail -n1)"
       DIFFERENCE_START_TIME=$(expr ${LATEST_IPV6NAT} - ${LATEST_STARTED} 2>/dev/null)
       if [[ "${DIFFERENCE_START_TIME}" -lt 30 ]]; then
         err_count=$(( ${err_count} + 1 ))
@@ -372,15 +494,16 @@ ipv6nat_checks() {
     progress "IPv6 NAT" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
     if [[ $? == 10 ]]; then
       diff_c=0
-      sleep 1
+      sleep 30
     else
       diff_c=0
-      sleep 3600
+      sleep 300
     fi
   done
   return 1
 }
 
+
 rspamd_checks() {
   err_count=0
   diff_c=0
@@ -388,15 +511,14 @@ rspamd_checks() {
   # Reduce error count by 2 after restarting an unhealthy container
   trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
   while [ ${err_count} -lt ${THRESHOLD} ]; do
-    cat /dev/null > /tmp/rspamd-mailcow
+    touch /tmp/rspamd-mailcow; echo "$(tail -50 /tmp/rspamd-mailcow)" > /tmp/rspamd-mailcow
     host_ip=$(get_container_ip rspamd-mailcow)
     err_c_cur=${err_count}
-    SCORE=$(/usr/bin/curl -s --data-binary @- --unix-socket /var/lib/rspamd/rspamd.sock http://rspamd/scan -d '
+    SCORE=$(echo 'To: null@localhost
-To: null@localhost
 From: watchdog@localhost
 
 Empty
-' | jq -rc .required_score)
+' | usr/bin/curl -s --data-binary @- --unix-socket /var/lib/rspamd/rspamd.sock http://rspamd/scan | jq -rc .required_score)
     if [[ ${SCORE} != "9999" ]]; then
       echo "Rspamd settings check failed" 2>> /tmp/rspamd-mailcow 1>&2
       err_count=$(( ${err_count} + 1))
@@ -406,13 +528,49 @@ Empty
     [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
     [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
     progress "Rspamd" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
-    diff_c=0
+    if [[ $? == 10 ]]; then
-    sleep $(( ( RANDOM % 30 )  + 10 ))
+      diff_c=0
+      sleep 1
+    else
+      diff_c=0
+      sleep $(( ( RANDOM % 60 ) + 20 ))
+    fi
   done
   return 1
 }
 
+olefy_checks() {
+  err_count=0
+  diff_c=0
+  THRESHOLD=5
+  # Reduce error count by 2 after restarting an unhealthy container
+  trap "[ ${err_count} -gt 1 ] && err_count=$(( ${err_count} - 2 ))" USR1
+  while [ ${err_count} -lt ${THRESHOLD} ]; do
+    touch /tmp/olefy-mailcow; echo "$(tail -50 /tmp/olefy-mailcow)" > /tmp/olefy-mailcow
+    host_ip=$(get_container_ip olefy-mailcow)
+    err_c_cur=${err_count}
+    /usr/lib/nagios/plugins/check_tcp -4 -H ${host_ip} -p 10055 2>> /tmp/olefy-mailcow 1>&2; err_count=$(( ${err_count} + $? ))
+    [ ${err_c_cur} -eq ${err_count} ] && [ ! $((${err_count} - 1)) -lt 0 ] && err_count=$((${err_count} - 1)) diff_c=1
+    [ ${err_c_cur} -ne ${err_count} ] && diff_c=$(( ${err_c_cur} - ${err_count} ))
+    progress "Olefy" ${THRESHOLD} $(( ${THRESHOLD} - ${err_count} )) ${diff_c}
+    if [[ $? == 10 ]]; then
+      diff_c=0
+      sleep 1
+    else
+      diff_c=0
+      sleep $(( ( RANDOM % 60 ) + 20 ))
+    fi
+  done
+  return 1
+}
+
+# Notify about start
+if [[ ! -z ${WATCHDOG_NOTIFY_EMAIL} ]]; then
+  mail_error "watchdog-mailcow" "Watchdog started monitoring mailcow."
+fi
+
 # Create watchdog agents
+
 (
 while true; do
   if ! nginx_checks; then
@@ -421,7 +579,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned nginx_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -431,7 +591,21 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned mysql_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
+
+(
+while true; do
+  if ! redis_checks; then
+    log_msg "Redis hit error limit"
+    echo redis-mailcow > /tmp/com_pipe
+  fi
+done
+) &
+PID=$!
+echo "Spawned redis_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -441,7 +615,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned phpfpm_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -451,7 +627,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned sogo_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 if [ ${CHECK_UNBOUND} -eq 1 ]; then
 (
@@ -462,7 +640,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned unbound_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 fi
 
 if [[ "${SKIP_CLAMD}" =~ ^([nN][oO]|[nN])+$ ]]; then
@@ -474,7 +654,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned clamd_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 fi
 
 (
@@ -485,7 +667,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned postfix_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -495,7 +679,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned dovecot_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -505,7 +691,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned rspamd_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -515,7 +703,45 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned ratelimit_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
+
+(
+while true; do
+  if ! fail2ban_checks; then
+    log_msg "Fail2ban hit error limit"
+    echo fail2ban > /tmp/com_pipe
+  fi
+done
+) &
+PID=$!
+echo "Spawned fail2ban_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
+
+#(
+#while true; do
+#  if ! olefy_checks; then
+#    log_msg "Olefy hit error limit"
+#    echo olefy-mailcow > /tmp/com_pipe
+#  fi
+#done
+#) &
+#PID=$!
+#echo "Spawned olefy_checks with PID ${PID}"
+#BACKGROUND_TASKS+=(${PID})
+
+(
+while true; do
+  if ! acme_checks; then
+    log_msg "ACME client hit error limit"
+    echo acme-mailcow > /tmp/com_pipe
+  fi
+done
+) &
+PID=$!
+echo "Spawned acme_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 (
 while true; do
@@ -525,7 +751,9 @@ while true; do
   fi
 done
 ) &
-BACKGROUND_TASKS+=($!)
+PID=$!
+echo "Spawned ipv6nat_checks with PID ${PID}"
+BACKGROUND_TASKS+=(${PID})
 
 # Monitor watchdog agents, stop script when agents fails and wait for respawn by Docker (restart:always:n)
 (
@@ -556,25 +784,43 @@ while true; do
 done
 ) &
 
-# Restart container when threshold limit reached
+# Actions when threshold limit is reached
 while true; do
   CONTAINER_ID=
   HAS_INITDB=
   read com_pipe_answer </tmp/com_pipe
+  if [ -s "/tmp/${com_pipe_answer}" ]; then
+    cat "/tmp/${com_pipe_answer}"
+  fi
   if [[ ${com_pipe_answer} == "ratelimit" ]]; then
     log_msg "At least one ratelimit was applied"
-    [[ ! -z ${WATCHDOG_NOTIFY_EMAIL} ]] && mail_error "${com_pipe_answer}" "No further information available."
+    [[ ! -z ${WATCHDOG_NOTIFY_EMAIL} ]] && mail_error "${com_pipe_answer}" "Please see mailcow UI logs for further information."
-  elif [[ ${com_pipe_answer} =~ .+-mailcow ]] || [[ ${com_pipe_answer} == "ipv6nat-mailcow" ]]; then
+  elif [[ ${com_pipe_answer} == "acme-mailcow" ]]; then
+    log_msg "acme-mailcow did not complete successfully"
+    [[ ! -z ${WATCHDOG_NOTIFY_EMAIL} ]] && mail_error "${com_pipe_answer}" "Please check acme-mailcow for further information."
+  elif [[ ${com_pipe_answer} == "fail2ban" ]]; then
+    F2B_RES=($(timeout 4s redis-cli -h redis-mailcow --raw GET F2B_RES 2> /dev/null))
+    if [[ ! -z "${F2B_RES}" ]]; then
+      redis-cli -h redis-mailcow DEL F2B_RES > /dev/null
+      host=
+      for host in "${F2B_RES[@]}"; do
+        log_msg "Banned ${host}"
+        rm /tmp/fail2ban 2> /dev/null
+        timeout 2s whois "${host}" > /tmp/fail2ban
+        [[ ! -z ${WATCHDOG_NOTIFY_EMAIL} ]] && [[ ${WATCHDOG_NOTIFY_BAN} =~ ^([yY][eE][sS]|[yY])+$ ]] && mail_error "${com_pipe_answer}" "IP ban: ${host}"
+      done
+    fi
+  elif [[ ${com_pipe_answer} =~ .+-mailcow ]]; then
     kill -STOP ${BACKGROUND_TASKS[*]}
-    sleep 3
+    sleep 10
     CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring | contains(\"${com_pipe_answer}\")) | .id")
     if [[ ! -z ${CONTAINER_ID} ]]; then
       if [[ "${com_pipe_answer}" == "php-fpm-mailcow" ]]; then
         HAS_INITDB=$(curl --silent --insecure -XPOST https://dockerapi/containers/${CONTAINER_ID}/top | jq '.msg.Processes[] | contains(["php -c /usr/local/etc/php -f /web/inc/init_db.inc.php"])' | grep true)
       fi
       S_RUNNING=$(($(date +%s) - $(curl --silent --insecure https://dockerapi/containers/${CONTAINER_ID}/json | jq .State.StartedAt | xargs -n1 date +%s -d)))
-      if [ ${S_RUNNING} -lt 120 ]; then
+      if [ ${S_RUNNING} -lt 360 ]; then
-        log_msg "Container is running for less than 120 seconds, skipping action..."
+        log_msg "Container is running for less than 360 seconds, skipping action..."
       elif [[ ! -z ${HAS_INITDB} ]]; then
         log_msg "Database is being initialized by php-fpm-mailcow, not restarting but delaying checks for a minute..."
         sleep 60
@@ -589,6 +835,7 @@ while true; do
       fi
     fi
     kill -CONT ${BACKGROUND_TASKS[*]}
+    sleep 1
     kill -USR1 ${BACKGROUND_TASKS[*]}
   fi
 done

data/assets/nextcloud/nextcloud.conf

@@ -75,7 +75,7 @@ server {
     deny all;
   }
 
-  location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
+  location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|oc[ms]-provider/.+)\.php(?:$|/) {
     fastcgi_split_path_info ^(.+\.php)(/.*)$;
     include fastcgi_params;
     fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
@@ -90,12 +90,12 @@ server {
     fastcgi_read_timeout 1200;
   }
 
-  location ~ ^/(?:updater|ocs-provider)(?:$|/) {
+  location ~ ^/(?:updater|oc[ms]-provider)(?:$|/) {
     try_files $uri/ =404;
     index index.php;
   }
 
-  location ~ \.(?:css|js|woff|svg|gif)$ {
+  location ~ \.(?:css|js|woff2?|svg|gif)$ {
     try_files $uri /index.php$uri$is_args$args;
     add_header Cache-Control "public, max-age=15778463";
     add_header X-Content-Type-Options nosniff;

data/assets/nextcloud/occ

@@ -1,2 +1,2 @@
 #!/bin/bash
-docker exec -it -u www-data $(docker ps -f name=php-fpm-mailcow -q) /web/nextcloud/occ ${@}
+docker exec -it -u www-data $(docker ps -f name=php-fpm-mailcow -q) php /web/nextcloud/occ ${@}

data/assets/nextcloud/site.nextcloud.custom

@@ -1,44 +0,0 @@
-  location ^~ /nextcloud {
-    location /nextcloud {
-      rewrite ^ /nextcloud/index.php$uri;
-    }
-    location ~ ^/nextcloud/(?:build|tests|config|lib|3rdparty|templates|data)/ {
-      deny all;
-    }
-    location ~ ^/nextcloud/(?:\.|autotest|occ|issue|indie|db_|console) {
-      deny all;
-    }
-    location ~ ^/nextcloud/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
-      fastcgi_split_path_info ^(.+\.php)(/.*)$;
-      include fastcgi_params;
-      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
-      fastcgi_param PATH_INFO $fastcgi_path_info;
-      fastcgi_param HTTPS on;
-      fastcgi_param modHeadersAvailable true;
-      fastcgi_param front_controller_active true;
-      fastcgi_pass phpfpm:9002;
-      fastcgi_intercept_errors on;
-      fastcgi_request_buffering off;
-      client_max_body_size 0;
-      fastcgi_read_timeout 1200;
-    }
-    location ~ ^/nextcloud/(?:updater|ocs-provider)(?:$|/) {
-      try_files $uri/ =404;
-      index index.php;
-    }
-    location ~ \.(?:css|js|woff|svg|gif)$ {
-      try_files $uri /nextcloud/index.php$uri$is_args$args;
-      add_header Cache-Control "public, max-age=15778463";
-      add_header X-Content-Type-Options nosniff;
-      add_header X-XSS-Protection "1; mode=block";
-      add_header X-Robots-Tag none;
-      add_header X-Download-Options noopen;
-      add_header X-Permitted-Cross-Domain-Policies none;
-      add_header X-Frame-Options "SAMEORIGIN";
-      access_log off;
-    }
-    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
-      try_files $uri /nextcloud/index.php$uri$is_args$args;
-      access_log off;
-    }
-  }

data/conf/dovecot/dovecot.conf

@@ -3,7 +3,7 @@
 # --------------------------------------------------------------------------
 # LDAP example:
 #passdb {
-#  args = /usr/local/etc/dovecot/ldap/passdb.conf
+#  args = /etc/dovecot/ldap/passdb.conf
 #  driver = ldap
 #}
 
@@ -20,7 +20,7 @@ disable_plaintext_auth = yes
 login_log_format_elements = "user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k"
 mail_home = /var/vmail/%d/%n
 mail_location = maildir:~/
-mail_plugins = </usr/local/etc/dovecot/mail_plugins
+mail_plugins = </etc/dovecot/mail_plugins
 mail_attachment_fs = crypt:set_prefix=mail_crypt_global:posix:
 mail_attachment_dir = /var/attachments
 mail_attachment_min_size = 128k
@@ -34,7 +34,7 @@ ssl_prefer_server_ciphers = yes
 ssl_cipher_list = ALL:!ADH:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL:!eNULL:!3DES:!MD5:!PSK:!DSS:!RC4:!SEED:!IDEA:+HIGH:+MEDIUM
 
 # Default in Dovecot 2.3
-ssl_options = no_compression
+ssl_options = no_compression no_ticket
 
 # New in Dovecot 2.3
 ssl_dh=</etc/ssl/mail/dhparams.pem
@@ -47,12 +47,12 @@ mail_shared_explicit_inbox = yes
 mail_prefetch_count = 30
 passdb {
   driver = passwd-file
-  args = /usr/local/etc/dovecot/dovecot-master.passwd
+  args = /etc/dovecot/dovecot-master.passwd
   master = yes
   pass = yes
 }
 passdb {
-  args = /usr/local/etc/dovecot/sql/dovecot-dict-sql-passdb.conf
+  args = /etc/dovecot/sql/dovecot-dict-sql-passdb.conf
   driver = sql
   result_success = return-ok
   result_failure = continue
@@ -60,7 +60,7 @@ passdb {
 }
 passdb {
   driver = passwd-file
-  args = /usr/local/etc/dovecot/dovecot-master.passwd
+  args = /etc/dovecot/dovecot-master.passwd
   skip = authenticated
 }
 # Set doveadm_password=your-secret-password in data/conf/dovecot/extra.conf (create if missing)
@@ -206,14 +206,6 @@ namespace inbox {
   }
   prefix =
 }
-namespace {
-    type = shared
-    separator = /
-    prefix = Shared/%%u/
-    location = maildir:%%h/:INDEX=~/Shared/%%u;CONTROL=~/Shared/%%u
-    subscriptions = no
-    list = children
-}
 protocols = imap sieve lmtp pop3
 service dict {
   unix_listener dict {
@@ -282,52 +274,53 @@ ssl_cert = </etc/ssl/mail/cert.pem
 ssl_key = </etc/ssl/mail/key.pem
 userdb {
   driver = passwd-file
-  args = /usr/local/etc/dovecot/dovecot-master.userdb
+  args = /etc/dovecot/dovecot-master.userdb
 }
 userdb {
-  args = /usr/local/etc/dovecot/sql/dovecot-dict-sql-userdb.conf
+  args = /etc/dovecot/sql/dovecot-dict-sql-userdb.conf
   driver = sql
   skip = found
 }
 protocol imap {
-  mail_plugins = </usr/local/etc/dovecot/mail_plugins_imap
+  mail_plugins = </etc/dovecot/mail_plugins_imap
   imap_metadata = yes
 }
 mail_attribute_dict = file:%h/dovecot-attributes
 protocol lmtp {
-  mail_plugins = </usr/local/etc/dovecot/mail_plugins_lmtp
+  mail_plugins = </etc/dovecot/mail_plugins_lmtp
-  auth_socket_path = /usr/local/var/run/dovecot/auth-master
+  auth_socket_path = /var/run/dovecot/auth-master
 }
 protocol sieve {
   managesieve_logout_format = bytes=%i/%o
 }
 plugin {
   # Allow "any" or "authenticated" to be used in ACLs
-  acl_anyone = </usr/local/etc/dovecot/acl_anyone
+  acl_anyone = </etc/dovecot/acl_anyone
   acl_shared_dict = file:/var/vmail/shared-mailboxes.db
   acl = vfile
   fts = solr
   fts_autoindex = yes
-  fts_solr = url=http://solr:8983/solr/dovecot/
+  fts_solr = url=http://solr:8983/solr/dovecot-fts/
   quota = dict:Userquota::proxy::sqlquota
   quota_rule2 = Trash:storage=+100%%
   sieve = /var/vmail/sieve/%u.sieve
   sieve_plugins = sieve_imapsieve sieve_extprograms
   sieve_vacation_send_from_recipient = yes
+  sieve_redirect_envelope_from = recipient
   # From elsewhere to Spam folder
   imapsieve_mailbox1_name = Junk
   imapsieve_mailbox1_causes = COPY
-  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve
+  imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve
   # END
   # From Spam folder to elsewhere
   imapsieve_mailbox2_name = *
   imapsieve_mailbox2_from = Junk
   imapsieve_mailbox2_causes = COPY
-  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
+  imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve
   # END
   quota_warning = storage=95%% quota-warning 95 %u
   quota_warning2 = storage=80%% quota-warning 80 %u
-  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
+  sieve_pipe_bin_dir = /usr/lib/dovecot/sieve
   sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
   sieve_extensions = +notify +imapflags +vacation-seconds
   sieve_max_script_size = 1M
@@ -338,9 +331,10 @@ plugin {
   sieve_vacation_min_period = 5s
   sieve_vacation_max_period = 0
   sieve_vacation_default_period = 60s
-  sieve_before = dict:proxy::sieve_before;name=active;bindir=/var/vmail/sieve_before_bindir
+  sieve_before = /var/vmail/sieve/global_sieve_before.sieve
+  sieve_before2 = dict:proxy::sieve_before;name=active;bindir=/var/vmail/sieve_before_bindir
   sieve_after = dict:proxy::sieve_after;name=active;bindir=/var/vmail/sieve_after_bindir
-  sieve_after2 = /var/vmail/sieve/global.sieve
+  sieve_after2 = /var/vmail/sieve/global_sieve_after.sieve
 
   # -- Global keys
   mail_crypt_global_private_key = </mail_crypt/ecprivkey.pem
@@ -363,9 +357,9 @@ service quota-warning {
   }
 }
 dict {
-  sqlquota = mysql:/usr/local/etc/dovecot/sql/dovecot-dict-sql-quota.conf
+  sqlquota = mysql:/etc/dovecot/sql/dovecot-dict-sql-quota.conf
-  sieve_after = mysql:/usr/local/etc/dovecot/sql/dovecot-dict-sql-sieve_after.conf
+  sieve_after = mysql:/etc/dovecot/sql/dovecot-dict-sql-sieve_after.conf
-  sieve_before = mysql:/usr/local/etc/dovecot/sql/dovecot-dict-sql-sieve_before.conf
+  sieve_before = mysql:/etc/dovecot/sql/dovecot-dict-sql-sieve_before.conf
 }
 remote 127.0.0.1 {
   disable_plaintext_auth = no
@@ -384,9 +378,12 @@ service stats {
   }
 }
 imap_max_line_length = 2 M
-auth_cache_verify_password_with_worker = yes
+#auth_cache_verify_password_with_worker = yes
-auth_cache_negative_ttl = 0
+#auth_cache_negative_ttl = 0
-auth_cache_ttl = 30 s
+#auth_cache_ttl = 30 s
-auth_cache_size = 2 M
+#auth_cache_size = 2 M
-!include_try /usr/local/etc/dovecot/extra.conf
+!include_try /etc/dovecot/extra.conf
+!include_try /etc/dovecot/sogo-sso.conf
+!include_try /etc/dovecot/shared_namespace.conf
 default_client_limit = 10400
+default_vsz_limit = 1024 M

data/conf/dovecot/global_sieve_after

@@ -1,3 +1,6 @@
+# global_sieve_after script
+# global_sieve_before -> user sieve_before (mailcow UI) -> user sieve_after (mailcow UI) -> global_sieve_after
+
 require "fileinto";
 require "mailbox";
 require "variables";

data/conf/dovecot/global_sieve_before

@@ -0,0 +1,2 @@
+# global_sieve_before script
+# global_sieve_before -> user sieve_before (mailcow UI) -> user sieve_after (mailcow UI) -> global_sieve_after

data/conf/nginx/site.conf

@@ -34,6 +34,7 @@ server {
 
   client_max_body_size 0;
 
+  listen 127.0.0.1:65510;
   include /etc/nginx/conf.d/listen_plain.active;
   include /etc/nginx/conf.d/listen_ssl.active;
   include /etc/nginx/conf.d/server_name.active;
@@ -142,7 +143,19 @@ server {
     try_files /autoconfig.php =404;
   }
 
+  # auth_request endpoint if ALLOW_ADMIN_EMAIL_LOGIN is set
+  location /sogo-auth-verify {
+    internal;
+    proxy_set_header  X-Original-URI $request_uri;
+    proxy_set_header  X-Real-IP $remote_addr;
+    proxy_set_header  Host $http_host;
+    proxy_set_header  Content-Length "";
+    proxy_pass        http://127.0.0.1:65510/sogo-auth;
+    proxy_pass_request_body off;
+  }
+
   location ^~ /Microsoft-Server-ActiveSync {
+    include /etc/nginx/conf.d/sogo_proxy_auth.active;
     include /etc/nginx/conf.d/sogo_eas.active;
     proxy_connect_timeout 4000;
     proxy_next_upstream timeout error;
@@ -165,6 +178,7 @@ server {
   }
 
   location ^~ /SOGo {
+    include /etc/nginx/conf.d/sogo_proxy_auth.active;
     include /etc/nginx/conf.d/sogo.active;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

data/conf/nginx/templates/sogo.auth_request.template.sh

@@ -0,0 +1,10 @@
+if printf "%s\n" "${ALLOW_ADMIN_EMAIL_LOGIN}" | grep -E '^([yY][eE][sS]|[yY])+$' >/dev/null; then
+    echo 'auth_request /sogo-auth-verify;
+auth_request_set $user $upstream_http_x_user;
+auth_request_set $auth $upstream_http_x_auth;
+auth_request_set $auth_type $upstream_http_x_auth_type;
+proxy_set_header x-webobjects-remote-user "$user";
+proxy_set_header Authorization "$auth";
+proxy_set_header x-webobjects-auth-type "$auth_type";
+'
+fi

data/conf/postfix/anonymize_headers.pcre

@@ -1,8 +1,11 @@
 if /^\s*Received:.*Authenticated sender.*\(Postcow\)/
-/^\s*Received:.*Authenticated sender:(.+)/
+#/^Received: from .*? \([\w-.]* \[.*?\]\)\s+\(Authenticated sender: (.+)\)\s+by.+\(Postcow\) with (E?SMTPS?A?) id ([A-F0-9]+).+;.*?/
-  REPLACE Received: from localhost (localhost [127.0.0.1]) (Authenticated sender:$1
+/^Received: from .*? \([\w-.]* \[.*?\]\)(.*|\n.*)\(Authenticated sender: (.+)\)\s+by.+\(Postcow\) with (.*)/
+  REPLACE Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with $2
 endif
 /^\s*X-Enigmail/        IGNORE
 /^\s*X-Mailer/          IGNORE
 /^\s*X-Originating-IP/  IGNORE
 /^\s*X-Forward/         IGNORE
+# Not removing UA by default, might be signed
+#/^\s*User-Agent/        IGNORE

data/conf/postfix/local_transport

@@ -0,0 +1,2 @@
+/watchdog@localhost$/   watchdog_discard:
+/localhost$/  local:

data/conf/postfix/main.cf

@@ -1,3 +1,6 @@
+# --------------------------------------------------------------------------
+# Please create a file "extra.cf" for persistent overrides to main.cf
+# --------------------------------------------------------------------------
 biff = no
 append_dot_mydomain = no
 smtpd_tls_cert_file = /etc/ssl/mail/cert.pem
@@ -6,7 +9,10 @@ smtpd_use_tls=yes
 smtpd_tls_received_header = yes
 smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
-smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
+smtpd_relay_restrictions = permit_mynetworks,
+  permit_sasl_authenticated,
+  defer_unauth_destination
+# alias maps are auto-generated in postfix.sh on startup
 alias_maps = hash:/etc/aliases
 alias_database = hash:/etc/aliases
 relayhost =
@@ -19,20 +25,53 @@ bounce_queue_lifetime = 1d
 broken_sasl_auth_clients = yes
 disable_vrfy_command = yes
 maximal_backoff_time = 1800s
-maximal_queue_lifetime = 1d
+maximal_queue_lifetime = 5d
+delay_warning_time = 4h
 message_size_limit = 104857600
 milter_default_action = accept
 milter_protocol = 6
 minimal_backoff_time = 300s
 plaintext_reject_code = 550
-postscreen_access_list = permit_mynetworks, cidr:/opt/postfix/conf/postscreen_access.cidr, tcp:127.0.0.1:10027
+postscreen_access_list = permit_mynetworks,
+  cidr:/opt/postfix/conf/postscreen_access.cidr,
+  tcp:127.0.0.1:10027
 postscreen_bare_newline_enable = no
 postscreen_blacklist_action = drop
 postscreen_cache_cleanup_interval = 24h
 postscreen_cache_map = proxy:btree:$data_directory/postscreen_cache
 postscreen_dnsbl_action = enforce
-postscreen_dnsbl_sites = b.barracudacentral.org=127.0.0.2*7 dnsbl.inps.de=127.0.0.2*7 bl.mailspike.net=127.0.0.2*5 bl.mailspike.net=127.0.0.[10;11;12]*4 dnsbl.sorbs.net=127.0.0.10*8 dnsbl.sorbs.net=127.0.0.5*6 dnsbl.sorbs.net=127.0.0.7*3 dnsbl.sorbs.net=127.0.0.8*2 dnsbl.sorbs.net=127.0.0.6*2 dnsbl.sorbs.net=127.0.0.9*2 zen.spamhaus.org=127.0.0.[10;11]*8 zen.spamhaus.org=127.0.0.[4..7]*6 zen.spamhaus.org=127.0.0.3*4 zen.spamhaus.org=127.0.0.2*3 hostkarma.junkemailfilter.com=127.0.0.2*3 hostkarma.junkemailfilter.com=127.0.0.4*1 hostkarma.junkemailfilter.com=127.0.1.2*1 wl.mailspike.net=127.0.0.[18;19;20]*-2 hostkarma.junkemailfilter.com=127.0.0.1*-2
+postscreen_dnsbl_sites = wl.mailspike.net=127.0.0.[18;19;20]*-2
-postscreen_dnsbl_threshold = 8
+  hostkarma.junkemailfilter.com=127.0.0.1*-2
+  list.dnswl.org=127.0.[0..255].0*-2
+  list.dnswl.org=127.0.[0..255].1*-4
+  list.dnswl.org=127.0.[0..255].2*-6
+  list.dnswl.org=127.0.[0..255].3*-8
+  ix.dnsbl.manitu.net*2
+  bl.spamcop.net*2
+  hostkarma.junkemailfilter.com=127.0.0.2*4
+  hostkarma.junkemailfilter.com=127.0.0.3*2
+  hostkarma.junkemailfilter.com=127.0.0.4*3
+  hostkarma.junkemailfilter.com=127.0.1.2*1
+  backscatter.spameatingmonkey.net*2
+  bl.ipv6.spameatingmonkey.net*2
+  bl.spameatingmonkey.net*2
+  b.barracudacentral.org=127.0.0.2*7
+  bl.mailspike.net=127.0.0.2*5
+  bl.mailspike.net=127.0.0.[10;11;12]*4
+  dnsbl.sorbs.net=127.0.0.10*8
+  dnsbl.sorbs.net=127.0.0.5*6
+  dnsbl.sorbs.net=127.0.0.7*3
+  dnsbl.sorbs.net=127.0.0.8*2
+  dnsbl.sorbs.net=127.0.0.6*2
+  dnsbl.sorbs.net=127.0.0.9*2
+  zen.spamhaus.org=127.0.0.[10;11]*8
+  zen.spamhaus.org=127.0.0.[4..7]*6
+  zen.spamhaus.org=127.0.0.3*4
+  zen.spamhaus.org=127.0.0.2*3
+  hostkarma.junkemailfilter.com=127.0.0.2*3
+  hostkarma.junkemailfilter.com=127.0.0.4*2
+  hostkarma.junkemailfilter.com=127.0.1.2*1
+postscreen_dnsbl_threshold = 5
 postscreen_dnsbl_ttl = 5m
 postscreen_greet_action = enforce
 postscreen_greet_banner = $smtpd_banner
@@ -40,16 +79,10 @@ postscreen_greet_ttl = 2d
 postscreen_greet_wait = 3s
 postscreen_non_smtp_command_enable = no
 postscreen_pipelining_enable = no
-proxy_read_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_sender_acl.cf,
+proxy_read_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_sasl_passwd_maps_transport_maps.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_tls_policy_override_maps.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_sender_dependent_default_transport_maps.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_transport_maps.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_sasl_passwd_maps_sender_dependent.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_sasl_passwd_maps_transport_maps.cf,
   proxy:mysql:/opt/postfix/conf/sql/mysql_tls_enforce_in_policy.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_sender_bcc_maps.cf,
+  $sender_dependent_default_transport_maps,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_recipient_bcc_maps.cf,
+  $smtp_tls_policy_maps,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_recipient_canonical_maps.cf,
   $local_recipient_maps,
   $mydestination,
   $virtual_alias_maps,
@@ -60,11 +93,14 @@ proxy_read_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_sender_acl.cf,
   $relay_domains,
   $canonical_maps,
   $sender_canonical_maps,
+  $sender_bcc_maps,
+  $recipient_bcc_maps,
   $recipient_canonical_maps,
   $relocated_maps,
   $transport_maps,
   $mynetworks,
-  $smtpd_sender_login_maps
+  $smtpd_sender_login_maps,
+  $smtp_sasl_password_maps
 queue_run_delay = 300s
 relay_domains = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_relay_domain_maps.cf
 relay_recipient_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_relay_recipient_maps.cf
@@ -81,33 +117,47 @@ smtpd_error_sleep_time = 10s
 smtpd_hard_error_limit = ${stress?1}${stress:5}
 smtpd_helo_required = yes
 smtpd_proxy_timeout = 600s
-smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, check_recipient_access proxy:mysql:/opt/postfix/conf/sql/mysql_tls_enforce_in_policy.cf, reject_invalid_helo_hostname, reject_unknown_reverse_client_hostname, reject_unauth_destination
+smtpd_recipient_restrictions = permit_sasl_authenticated,
+  permit_mynetworks,
+  check_recipient_access proxy:mysql:/opt/postfix/conf/sql/mysql_tls_enforce_in_policy.cf,
+  reject_invalid_helo_hostname,
+  reject_unknown_reverse_client_hostname,
+  reject_unauth_destination
 smtpd_sasl_auth_enable = yes
 smtpd_sasl_authenticated_header = yes
 smtpd_sasl_path = inet:dovecot:10001
 smtpd_sasl_type = dovecot
 smtpd_sender_login_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_sender_acl.cf
-smtpd_sender_restrictions = reject_authenticated_sender_login_mismatch, permit_mynetworks, permit_sasl_authenticated, reject_unlisted_sender, reject_unknown_sender_domain
+smtpd_sender_restrictions = reject_authenticated_sender_login_mismatch,
+  permit_mynetworks,
+  permit_sasl_authenticated,
+  reject_unlisted_sender,
+  reject_unknown_sender_domain
 smtpd_soft_error_limit = 3
 smtpd_tls_auth_only = yes
 smtpd_tls_dh1024_param_file = /etc/ssl/mail/dhparams.pem
 smtpd_tls_eecdh_grade = auto
 smtpd_tls_exclude_ciphers = ECDHE-RSA-RC4-SHA, RC4, aNULL, DES-CBC3-SHA, ECDHE-RSA-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA
 smtpd_tls_loglevel = 1
-smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
+
+# Mandatory protocols and ciphers are used when a connections is enforced to use TLS
+# Does _not_ apply to enforced incoming TLS settings per mailbox
+smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
+lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
+smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
+smtpd_tls_mandatory_ciphers = high
+
 smtp_tls_protocols = !SSLv2, !SSLv3
-lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3
+lmtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
-lmtp_tls_protocols = !SSLv2, !SSLv2, !SSLv3
-smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
 smtpd_tls_protocols = !SSLv2, !SSLv3
+
 smtpd_tls_security_level = may
 tls_preempt_cipherlist = yes
 tls_ssl_options = NO_COMPRESSION
-smtpd_tls_mandatory_ciphers = high
 virtual_alias_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_alias_maps.cf,
+  proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_resource_maps.cf,
   proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_spamalias_maps.cf,
-  proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_alias_domain_maps.cf,
+  proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_alias_domain_maps.cf
-  proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_alias_domain_catchall_maps.cf
 virtual_gid_maps = static:5000
 virtual_mailbox_base = /var/vmail/
 virtual_mailbox_domains = proxy:mysql:/opt/postfix/conf/sql/mysql_virtual_domains_maps.cf
@@ -123,7 +173,6 @@ smtpd_milters = inet:rspamd:9900
 non_smtpd_milters = inet:rspamd:9900
 milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
 mydestination = localhost.localdomain, localhost
-#content_filter=zeyple
 # Prefere IPv4, useful for v4-only envs
 smtp_address_preference = ipv4
 smtp_sender_dependent_authentication = yes
@@ -134,5 +183,14 @@ smtp_sasl_mechanism_filter = plain, login
 smtp_tls_policy_maps=proxy:mysql:/opt/postfix/conf/sql/mysql_tls_policy_override_maps.cf
 smtp_header_checks = pcre:/opt/postfix/conf/anonymize_headers.pcre
 mail_name = Postcow
-transport_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_transport_maps.cf
+# local_transport map catches local destinations and prevents routing local dests when the next map would route "*"
+transport_maps = pcre:/opt/postfix/conf/local_transport,
+  proxy:mysql:/opt/postfix/conf/sql/mysql_transport_maps.cf
 smtp_sasl_auth_soft_bounce = no
+postscreen_discard_ehlo_keywords = silent-discard, dsn
+compatibility_level = 2
+smtputf8_enable = no
+
+# DO NOT EDIT ANYTHING BELOW #
+# User overrides #
+

data/conf/postfix/master.cf

@@ -1,29 +1,47 @@
+# inter-mx with postscreen on 25/tcp
 smtp       inet  n       -       n       -       1       postscreen
 smtpd      pass  -       -       n       -       -       smtpd
   -o smtpd_helo_restrictions=permit_mynetworks,reject_non_fqdn_helo_hostname
   -o smtpd_sasl_auth_enable=no
+  -o smtpd_sender_restrictions=permit_mynetworks,reject_unlisted_sender,reject_unknown_sender_domain
+
+# smtpd tls-wrapped (smtps) on 465/tcp
 smtps    inet  n       -       n       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
   -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
+  -o smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
   -o tls_preempt_cipherlist=yes
+
+# smtpd with starttls on 587/tcp
 submission inet n       -       n       -       -       smtpd
   -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
   -o smtpd_enforce_tls=yes
   -o smtpd_tls_security_level=encrypt
+  -o smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
   -o tls_preempt_cipherlist=yes
+
+# used by SOGo
+# smtpd_sender_restrictions should match main.cf, but with check_sasl_access prepended for login-as-mailbox-user function
 588 inet n      -       n       -       -       smtpd
   -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
   -o smtpd_tls_auth_only=no
   -o smtpd_sender_restrictions=check_sasl_access,regexp:/opt/postfix/conf/allow_mailcow_local.regexp,reject_authenticated_sender_login_mismatch,permit_mynetworks,permit_sasl_authenticated,reject_unlisted_sender,reject_unknown_sender_domain
+
+# used to reinject quarantine mails
 590 inet n      -       n       -       -       smtpd
   -o smtpd_client_restrictions=permit_mynetworks,reject
   -o smtpd_tls_auth_only=no
   -o smtpd_milters=
   -o non_smtpd_milters=
+
+# enforced smtp connector
 smtp_enforced_tls      unix  -       -       n       -       -       smtp
   -o smtp_tls_security_level=encrypt
   -o syslog_name=enforced-tls-smtp
   -o smtp_delivery_status_filter=pcre:/opt/postfix/conf/smtp_dsn_filter
+
+# smtp connector used, when a transport map matched
+# this helps to have different sasl maps than we have with sender dependent transport maps
 smtp_via_transport_maps      unix  -       -       n       -       -       smtp
   -o smtp_sasl_password_maps=proxy:mysql:/opt/postfix/conf/sql/mysql_sasl_passwd_maps_transport_maps.cf
 
@@ -55,25 +73,12 @@ scache     unix  -       -       n       -       1       scache
 maildrop   unix  -       n       n       -       -       pipe flags=DRhu
     user=vmail argv=/usr/bin/maildrop -d ${recipient}
 
-# start zeyple
-zeyple    unix  -       n       n       -       -       pipe
-  user=zeyple argv=/usr/local/bin/zeyple.py ${recipient}
-127.0.0.1:10026 inet  n       -       n       -       10      smtpd
-  -o content_filter=
-  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
-  -o smtpd_helo_restrictions=
-  -o smtpd_client_restrictions=
-  -o smtpd_sender_restrictions=
-  -o smtpd_recipient_restrictions=permit_mynetworks,reject
-  -o mynetworks=127.0.0.0/8
-  -o smtpd_authorized_xforward_hosts=127.0.0.0/8
-# end zeyple
-
 # start whitelist_fwd
 127.0.0.1:10027 inet n n n - 0 spawn user=nobody argv=/usr/local/bin/whitelist_forwardinghosts.sh
 # end whitelist_fwd
 
 # start watchdog-specific
+# logs to local7 (hidden)
 589 inet n      -       n       -       -       smtpd
   -o smtpd_client_restrictions=permit_mynetworks,reject
   -o syslog_name=watchdog

data/conf/rspamd/custom/bad_words.map

@@ -0,0 +1,44 @@
+/\ssex\s/i
+/\svagina\s/i
+/\serotic\s/i
+/\serection\s/i
+/\ssexy\s/i
+/\spenis\s/i
+/\sass\s/i
+/\sviagra\s/i
+/\stits\s/i
+/\stitty\s/i
+/\stitties\s/i
+/\scum\s/i
+/\ssperm\s/i
+/\sslut\s/i
+/\sporn\s/i
+/\scock\s/i
+/\spharma\s/i
+/\spharmacy\s/i
+/\sseo\s/i
+/\smarketing\s/i
+/\sjackpot\s/i
+/\slotto\s/i
+/\slottery\s/i
+/pillenversand/i
+/\skredithilfe\s/i
+/\skapital\s/i
+/\skrankenversicherung\s/i
+/bitcoin/i
+/pädophil/i
+/paedophil/i
+/freiberufler/i
+/unternehmer/i
+/masturbieren/i
+/trojaner/i
+/malware/i
+/\sscooter\s/i
+/\sescooter\s/i
+/\se-scooter\s/i
+/testost/i
+/\spotenz\s/i
+/potenzmittel/i
+/rezeptfrei/i
+/apotheke/i
+/web\sdevelopment/i

data/conf/rspamd/custom/fishy_tlds.map

@@ -0,0 +1,66 @@
+/.+\.accountant$/i
+/.+\.art$/i
+/.+\.asia$/i
+/.+\.bid$/i
+/.+\.biz$/i
+/.+\.care$/i
+/.+\.cf$/i
+/.+\.cl$/i
+/.+\.click$/i
+/.+\.cloud$/i
+/.+\.co$/i
+/.+\.construction$/i
+/.+\.country$/i
+/.+\.cricket$/i
+/.+\.date$/i
+/.+\.desi$/i
+/.+\.download$/i
+/.+\.estate$/i
+/.+\.faith$/i
+/.+\.fit$/i
+/.+\.flights$/i
+/.+\.ga$/i
+/.+\.gdn$/i
+/.+\.gq$/i
+/.+\.guru$/i
+/.+\.icu$/i
+/.+\.id$/i
+/.+\.info$/i
+/.+\.in.net$/i
+/.+\.ir$/i
+/.+\.jetzt$/i
+/.+\.kim$/i
+/.+\.life$/i
+/.+\.link$/i
+/.+\.loan$/i
+/.+\.mk$/i
+/.+\.ml$/i
+/.+\.ninja$/i
+/.+\.online$/i
+/.+\.ooo$/i
+/.+\.party$/i
+/.+\.pro$/i
+/.+\.ps$/i
+/.+\.pw$/i
+/.+\.racing$/i
+/.+\.review$/i
+/.+\.rocks$/i
+/.+\.ryukyu$/i
+/.+\.science$/i
+/.+\.site$/i
+/.+\.space$/i
+/.+\.stream$/i
+/.+\.sucks$/i
+/.+\.tk$/i
+/.+\.top$/i
+/.+\.topica\.com$/i
+/.+\.town$/i
+/.+\.trade$/i
+/.+\.uno$/i
+/.+\.vip$/i
+/.+\.webcam$/i
+/.+\.website$/i
+/.+\.win$/i
+/.+\.work$/i
+/.+\.world$/i
+/.+\.xyz$/i

data/conf/rspamd/custom/ip_wl.map

@@ -0,0 +1,4 @@
+# IP whitelist
+# 127.0.0.1
+# 1.2.3.4
+# ...

data/conf/rspamd/dynmaps/settings.php

@@ -6,6 +6,8 @@ then any of these will trigger the rule. If a rule is triggered then no more rul
 */
 header('Content-Type: text/plain');
 require_once "vars.inc.php";
+// Getting headers sent by the client.
+//$headers = apache_request_headers();
 
 ini_set('error_reporting', 0);
 
@@ -25,6 +27,23 @@ catch (PDOException $e) {
   exit;
 }
 
+// Check if db changed and return header
+/*$stmt = $pdo->prepare("SELECT UNIX_TIMESTAMP(UPDATE_TIME) AS `db_update_time` FROM information_schema.tables
+  WHERE `TABLE_NAME` = 'filterconf'
+    AND TABLE_SCHEMA = :dbname;");
+$stmt->execute(array(
+  ':dbname' => $database_name
+));
+$db_update_time = $stmt->fetch(PDO::FETCH_ASSOC)['db_update_time'];
+
+if (isset($headers['If-Modified-Since']) && (strtotime($headers['If-Modified-Since']) == $db_update_time)) {
+  header('Last-Modified: '.gmdate('D, d M Y H:i:s', $db_update_time).' GMT', true, 304);
+  exit;
+} else {
+  header('Last-Modified: '.gmdate('D, d M Y H:i:s', $db_update_time).' GMT', true, 200);
+}
+*/
+
 function parse_email($email) {
   if (!filter_var($email, FILTER_VALIDATE_EMAIL)) return false;
   $a = strrpos($email, '@');
@@ -43,7 +62,9 @@ function wl_by_sogo() {
       if (!filter_var($contact, FILTER_VALIDATE_EMAIL)) {
         continue;
       }
-      $rcpt[$row['user']][] = '/^' . str_replace('/', '\/', $contact) . '$/i';
+      // Explicit from, no mime_from, no regex - envelope must match
+      // mailcow white and blacklists also cover mime_from
+      $rcpt[$row['user']][] = str_replace('/', '\/', $contact);
     }
   }
   return $rcpt;
@@ -67,7 +88,7 @@ function ucl_rcpts($object, $type) {
       if (!empty($local) && !empty($domain)) {
         $rcpt[] = '/^' . str_replace('/', '\/', $local) . '[+].*' . str_replace('/', '\/', $domain) . '$/i';
       }
-      $rcpt[] = '/^' . str_replace('/', '\/', $row['address']) . '$/i';
+      $rcpt[] = str_replace('/', '\/', $row['address']);
     }
     // Aliases by alias domains
     $stmt = $pdo->prepare("SELECT CONCAT(`local_part`, '@', `alias_domain`.`alias_domain`) AS `alias` FROM `mailbox` 
@@ -85,7 +106,7 @@ function ucl_rcpts($object, $type) {
         if (!empty($local) && !empty($domain)) {
           $rcpt[] = '/^' . str_replace('/', '\/', $local) . '[+].*' . str_replace('/', '\/', $domain) . '$/i';
         }
-      $rcpt[] = '/^' . str_replace('/', '\/', $row['alias']) . '$/i';
+        $rcpt[] = str_replace('/', '\/', $row['alias']);
       }
     }
   }
@@ -107,8 +128,8 @@ function ucl_rcpts($object, $type) {
 settings {
   watchdog {
     priority = 10;
-    rcpt = "/null@localhost/i";
+    rcpt_mime = "/null@localhost/i";
-    from = "/watchdog@localhost/i";
+    from_mime = "/watchdog@localhost/i";
     apply "default" {
       actions {
         reject = 9999.0;
@@ -199,12 +220,13 @@ while ($row = array_shift($rows)) {
 ?>
   whitelist_<?=$username_sane;?> {
 <?php
+  $list_items = array();
   $stmt = $pdo->prepare("SELECT `value` FROM `filterconf`
     WHERE `object`= :object
       AND `option` = 'whitelist_from'");
   $stmt->execute(array(':object' => $row['object']));
   $list_items = $stmt->fetchAll(PDO::FETCH_ASSOC);
-  while ($item = array_shift($list_items)) {
+  foreach ($list_items as $item) {
 ?>
     from = "/<?='^' . str_replace('\*', '.*', preg_quote($item['value'], '/')) . '$' ;?>/i";
 <?php
@@ -237,24 +259,13 @@ while ($row = array_shift($rows)) {
       "MAILCOW_WHITE"
     ]
   }
-  whitelist_header_<?=$username_sane;?> {
+  whitelist_mime_<?=$username_sane;?> {
 <?php
-  $header_from = array();
+  foreach ($list_items as $item) {
-  $stmt = $pdo->prepare("SELECT `value` FROM `filterconf`
-    WHERE `object`= :object
-      AND `option` = 'whitelist_from'");
-  $stmt->execute(array(':object' => $row['object']));
-  $list_items = $stmt->fetchAll(PDO::FETCH_ASSOC);
 ?>
-    header = {
+    from_mime = "/<?='^' . str_replace('\*', '.*', preg_quote($item['value'], '/')) . '$' ;?>/i";
 <?php
-  while ($item = array_shift($list_items)) {
-    $header_from[] = str_replace('\*', '.*', preg_quote($item['value'], '/'));
   }
-?>
-      "From" = "/(<?=implode('|', $header_from);?>)/i";
-    }
-<?php
   if (!filter_var(trim($row['object']), FILTER_VALIDATE_EMAIL)) {
 ?>
     priority = 5;
@@ -297,13 +308,13 @@ while ($row = array_shift($rows)) {
 ?>
   blacklist_<?=$username_sane;?> {
 <?php
-  $items[] = array();
+  $list_items = array();
   $stmt = $pdo->prepare("SELECT `value` FROM `filterconf`
     WHERE `object`= :object
       AND `option` = 'blacklist_from'");
   $stmt->execute(array(':object' => $row['object']));
   $list_items = $stmt->fetchAll(PDO::FETCH_ASSOC);
-  while ($item = array_shift($list_items)) {
+  foreach ($list_items as $item) {
 ?>
     from = "/<?='^' . str_replace('\*', '.*', preg_quote($item['value'], '/')) . '$' ;?>/i";
 <?php
@@ -338,22 +349,11 @@ while ($row = array_shift($rows)) {
   }
   blacklist_header_<?=$username_sane;?> {
 <?php
-  $header_from = array();
+  foreach ($list_items as $item) {
-  $stmt = $pdo->prepare("SELECT `value` FROM `filterconf`
-    WHERE `object`= :object
-      AND `option` = 'blacklist_from'");
-  $stmt->execute(array(':object' => $row['object']));
-  $list_items = $stmt->fetchAll(PDO::FETCH_ASSOC);
 ?>
-    header = {
+    from_mime = "/<?='^' . str_replace('\*', '.*', preg_quote($item['value'], '/')) . '$' ;?>/i";
 <?php
-  while ($item = array_shift($list_items)) {
-    $header_from[] = str_replace('\*', '.*', preg_quote($item['value'], '/'));
   }
-?>
-      "From" = "/(<?=implode('|', $header_from);?>)/i";
-    }
-<?php
   if (!filter_var(trim($row['object']), FILTER_VALIDATE_EMAIL)) {
 ?>
     priority = 5;

data/conf/rspamd/local.d/arc.conf

@@ -28,3 +28,5 @@ use_redis = true;
 key_prefix = "DKIM_PRIV_KEYS";
 # Selector map
 selector_prefix = "DKIM_SELECTORS";
+sign_inbound = true;
+use_domain_sign_inbound = "recipient";

data/conf/rspamd/local.d/composites.conf

@@ -16,3 +16,17 @@ SOGO_CONTACT_EXCLUDE_FWD_HOST {
 SOGO_CONTACT_SPOOFED {
   expression = "(R_SPF_PERMFAIL | R_SPF_SOFTFAIL | R_SPF_FAIL) & ~SOGO_CONTACT";
 }
+SPOOFED_UNAUTH {
+  expression = "!MAILCOW_AUTH & !MAILCOW_WHITE & !R_SPF_ALLOW & !DMARC_POLICY_ALLOW & !ARC_ALLOW & !SIEVE_HOST & MAILCOW_DOMAIN_HEADER_FROM";
+  score = 5.0;
+}
+# Only apply to inbound unauthed and not whitelisted
+OLEFY_MACRO {
+  expression = "!MAILCOW_AUTH & !MAILCOW_WHITE & OLETOOLS";
+  score = 20.0;
+  policy = "remove_weight";
+}
+BAD_WORD_BAD_TLD {
+  expression = "FISHY_TLD & BAD_WORDS"
+  score = 10.0;
+}

data/conf/rspamd/local.d/external_services.conf

@@ -0,0 +1,7 @@
+oletools {
+  # default olefy settings
+  servers = "olefy:10055";
+  # needs to be set explicitly for Rspamd < 1.9.5
+  scan_mime_parts = true;
+  # mime-part regex matching in content-type or filename
+}

data/conf/rspamd/local.d/metadata_exporter.conf

@@ -20,7 +20,7 @@ return function(task)
   if ratelimited then
     return true
   end
-  return
+  return false
 end
 EOD;
 }

data/conf/rspamd/local.d/milter_headers.conf

@@ -13,6 +13,7 @@ routines {
   authentication-results {
     header = "Authentication-Results";
     remove = 1;
+    add_smtp_user = false;
     spf_symbols {
       pass = "R_SPF_ALLOW";
       fail = "R_SPF_FAIL";

data/conf/rspamd/local.d/multimap.conf

@@ -83,3 +83,39 @@ GLOBAL_RCPT_BL {
   prefilter = true;
   action = "reject";
 }
+
+SIEVE_HOST {
+  type = "ip";
+  map = "$LOCAL_CONFDIR/custom/dovecot_trusted.map";
+  symbols_set = ["SIEVE_HOST"];
+}
+
+MAILCOW_DOMAIN_HEADER_FROM { 
+  type = "header";  
+  header = "from";  
+  filter = "email:domain";  
+  map = "redis://DOMAIN_MAP"; 
+}
+
+IP_WHITELIST {
+  type = "ip";
+  map = "$LOCAL_CONFDIR/custom/ip_wl.map";
+  prefilter = "true";
+  action = "accept";
+}
+
+FISHY_TLD {
+  type = "from";
+  filter = "email:domain";
+  map = "${LOCAL_CONFDIR}/custom/fishy_tlds.map";
+  regexp = true;
+  score = 0.1;
+}
+
+BAD_WORDS {
+  type = "content";
+  filter = "text";
+  map = "${LOCAL_CONFDIR}/custom/bad_words.map";
+  regexp = true;
+  score = 0.1;
+}

data/conf/rspamd/local.d/policies_group.conf

@@ -11,7 +11,13 @@ symbols = {
     "R_DKIM_REJECT" {
         score = 10.0;
     }
-    "R_DKIM_PERMFAIL" {
+    "DMARC_POLICY_REJECT" {
-        score = 10.0;
+        weight = 20.0;
+    }
+    "DMARC_POLICY_QUARANTINE" {
+        weight = 10.0;
+    }
+    "DMARC_POLICY_SOFTFAIL" {
+        weight = 2.0;
     }
 }

data/conf/rspamd/local.d/rbl.conf

@@ -0,0 +1,10 @@
+rbls {
+  uceprotect1 {
+    symbol = "RBL_UCEPROTECT_LEVEL1";
+    rbl = "dnsbl-1.uceprotect.net";
+  }
+  uceprotect2 {
+    symbol = "RBL_UCEPROTECT_LEVEL2";
+    rbl = "dnsbl-2.uceprotect.net";
+  }
+}

data/conf/rspamd/local.d/rbl_group.conf

@@ -0,0 +1,8 @@
+symbols = {
+  "RBL_UCEPROTECT_LEVEL1" {
+    score = 3.5;
+  }
+  "RBL_UCEPROTECT_LEVEL2" {
+    score = 1.5;
+  }
+}

data/conf/rspamd/local.d/rspamd.conf.local

@@ -1,16 +0,0 @@
-# rspamd.conf.local
-
-worker "fuzzy" {
-  # Socket to listen on (UDP and TCP from rspamd 1.3)
-  bind_socket = "*:11445";
-  allow_update = ["127.0.0.1", "::1"];
-  # Number of processes to serve this storage (useful for read scaling)
-  count = 2;
-  # Backend ("sqlite" or "redis" - default "sqlite")
-  backend = "redis";
-  # Hashes storage time (3 months)
-  expire = 90d;
-  # Synchronize updates to the storage each minute
-  sync = 1min;
-}
-

data/conf/rspamd/local.d/spamassassin.conf

@@ -0,0 +1 @@
+ruleset = "/etc/rspamd/custom/sa-rules";

data/conf/rspamd/local.d/statistics_group.conf

@@ -1,10 +1,10 @@
 symbols = {
     "BAYES_SPAM" {
-        weight = 8.5;
+        weight = 2.5;
         description = "Message probably spam, probability: ";
     }
     "BAYES_HAM" {
-        weight = -12.5;
+        weight = -10.5;
         description = "Message probably ham, probability: ";
     }
 }

data/conf/rspamd/meta_exporter/pipe.php

@@ -84,6 +84,9 @@ $rcpt_final_mailboxes = array();
 
 // Loop through all rcpts
 foreach (json_decode($rcpts, true) as $rcpt) {
+  // Remove tag
+  $rcpt = preg_replace('/^(.*?)\+.*(@.*)$/', '$1$2', $rcpt);
+  
   // Break rcpt into local part and domain part
   $parsed_rcpt = parse_email($rcpt);
   
@@ -128,6 +131,14 @@ foreach (json_decode($rcpts, true) as $rcpt) {
       ));
       $gotos = $stmt->fetch(PDO::FETCH_ASSOC)['goto'];
     }
+    if (empty($gotos)) {
+      $stmt = $pdo->prepare("SELECT `target_domain` FROM `alias_domain` WHERE `alias_domain` = :rcpt AND `active` = '1'");
+      $stmt->execute(array(':rcpt' => $parsed_rcpt['domain']));
+      $goto_branch = $stmt->fetch(PDO::FETCH_ASSOC)['target_domain'];
+      if ($goto_branch) {
+        $gotos = $parsed_rcpt['local'] . '@' . $goto_branch;
+      }
+    }
     $gotos_array = explode(',', $gotos);
 
     $loop_c = 0;
@@ -156,8 +167,18 @@ foreach (json_decode($rcpts, true) as $rcpt) {
             $stmt = $pdo->prepare("SELECT `goto` FROM `alias` WHERE `address` = :goto AND `active` = '1'");
             $stmt->execute(array(':goto' => $goto));
             $goto_branch = $stmt->fetch(PDO::FETCH_ASSOC)['goto'];
-            error_log("QUARANTINE: quarantine pipe: goto address " . $goto . " is a alias branch for " . $goto_branch);
+            if ($goto_branch) {
-            $goto_branch_array = explode(',', $goto_branch);
+              error_log("QUARANTINE: quarantine pipe: goto address " . $goto . " is a alias branch for " . $goto_branch);
+              $goto_branch_array = explode(',', $goto_branch);
+            } else {
+              $stmt = $pdo->prepare("SELECT `target_domain` FROM `alias_domain` WHERE `alias_domain` = :domain AND `active` AND '1'");
+              $stmt->execute(array(':domain' => $parsed_goto['domain']));
+              $goto_branch = $stmt->fetch(PDO::FETCH_ASSOC)['target_domain'];
+              if ($goto_branch) {
+                error_log("QUARANTINE: quarantine pipe: goto domain " . $parsed_gto['domain'] . " is a domain alias branch for " . $goto_branch);
+                $goto_branch_array = array($parsed_gto['local'] . '@' . $goto_branch);
+              }
+            }
           }
         }
         // goto item was processed, unset

data/conf/rspamd/override.d/ratelimit.conf

@@ -1,8 +1,8 @@
 rates {
     # Format: "1 / 1h" or "20 / 1m" etc. - global ratelimits are disabled by default
-    to = "45 / 1m";
+    to = "100 / 1s";
-    to_ip = "360 / 1m";
+    to_ip = "100 / 1s";
-    to_ip_from = "180 / 1m";
+    to_ip_from = "100 / 1s";
     bounce_to = "100 / 1s";
     bounce_to_ip = "100 / 1s";
 }

data/conf/rspamd/override.d/worker-controller-password.inc

@@ -1 +0,0 @@
-# Placeholder

data/conf/rspamd/override.d/worker-fuzzy.inc

@@ -0,0 +1,12 @@
+# Socket to listen on (UDP and TCP from rspamd 1.3)
+bind_socket = "*:11445";
+allow_update = ["127.0.0.1", "::1"];
+# Number of processes to serve this storage (useful for read scaling)
+count = 2;
+# Backend ("sqlite" or "redis" - default "sqlite")
+backend = "redis";
+# Hashes storage time (3 months)
+expire = 90d;
+# Synchronize updates to the storage each minute
+sync = 1min;
+

data/conf/rspamd/override.d/worker-proxy.inc

@@ -1,6 +1,6 @@
 bind_socket = "rspamd:9900";
 milter = true;
-upstream {
+upstream "local" {
   name = "localhost";
   default = true;
   hosts = "rspamd:11333"

data/conf/rspamd/plugins.d/README.md

@@ -0,0 +1 @@
+This is where you should copy any rspamd custom module

data/conf/rspamd/rspamd.conf.local

@@ -0,0 +1 @@
+# rspamd.conf.local

data/conf/rspamd/rspamd.conf.override

@@ -0,0 +1,2 @@
+# rspamd.conf.override
+

data/conf/sogo/sogo.conf

@@ -26,7 +26,6 @@
     //  (domain3.tld, domain2.tld)
     // );
 
-    SOGoIMAPServer = "imap://dovecot:143/?tls=YES";
     SOGoSieveServer = "sieve://dovecot:4190/?tls=YES";
     SOGoSMTPServer = "postfix:588";
     WOPort = "0.0.0.0:20000";

data/conf/unbound/unbound.conf

@@ -32,6 +32,7 @@ server:
   hide-version: yes
   max-udp-size: 4096
   msg-buffer-size: 65552
+  unwanted-reply-threshold: 10000
 
 remote-control:
     control-enable: yes

data/web/admin.php

@@ -5,6 +5,9 @@ if (isset($_SESSION['mailcow_cc_role']) && $_SESSION['mailcow_cc_role'] == "admi
 require_once $_SERVER['DOCUMENT_ROOT'] . '/inc/header.inc.php';
 $_SESSION['return_to'] = $_SERVER['REQUEST_URI'];
 $tfa_data = get_tfa();
+if (!isset($_SESSION['gal']) && $license_cache = $redis->Get('LICENSE_STATUS_CACHE')) {
+  $_SESSION['gal'] = json_decode($license_cache, true);
+}
 ?>
 <div class="container">
 
@@ -76,8 +79,40 @@ $tfa_data = get_tfa();
             </select>
           </div>
         </div>
-        <legend data-target="#api" style="margin-top:40px;cursor:pointer" class="arrow-toggle" unselectable="on" data-toggle="collapse">
+
-          <span style="font-size:12px" class="arrow rotate glyphicon glyphicon-menu-down"></span> API (experimental, work in progress)
+        <legend data-target="#license" class="arrow-toggle" unselectable="on" data-toggle="collapse">
+          <span style="font-size:12px" class="arrow rotate glyphicon glyphicon-menu-down"></span> <?=$lang['admin']['guid_and_license'];?>
+        </legend>
+        <div id="license" class="collapse in">
+        <form class="form-horizontal" autocapitalize="none" autocorrect="off" role="form" method="post">
+          <div class="form-group">
+            <label class="control-label col-sm-3" for="guid"><?=$lang['admin']['guid'];?>:</label>
+            <div class="col-sm-9">
+              <div class="input-group">
+                <span class="input-group-addon">
+                  <span class="glyphicon <?=(isset($_SESSION['gal']['valid']) && $_SESSION['gal']['valid'] === "true") ? 'glyphicon-heart text-danger' : 'glyphicon-remove';?>" aria-hidden="true"></span>
+                </span>
+                <input type="text" id="guid" class="form-control" value="<?=license('guid');?>" readonly>
+              </div>
+              <p class="help-block">
+                <?=$lang['admin']['customer_id'];?>: <?=(isset($_SESSION['gal']['c'])) ? $_SESSION['gal']['c'] : '?';?> -
+                <?=$lang['admin']['service_id'];?>: <?=(isset($_SESSION['gal']['s'])) ? $_SESSION['gal']['s'] : '?';?>
+              </p>
+            </div>
+          </div>
+          <div class="form-group">
+            <div class="col-sm-offset-3 col-sm-9">
+              <p class="help-block"><?=$lang['admin']['license_info'];?></p>
+              <div class="btn-group">
+                <button class="btn btn-sm btn-success" name="license_validate_now" type="submit" href="#"><?=$lang['admin']['validate_license_now'];?></button>
+              </div>
+            </div>
+          </div>
+        </form>
+        </div>
+
+        <legend data-target="#api" class="arrow-toggle" unselectable="on" data-toggle="collapse">
+          <span style="font-size:12px" class="arrow rotate glyphicon glyphicon-menu-down"></span> API
         </legend>
         <?php
         $api = admin_api('get');
@@ -105,6 +140,7 @@ $tfa_data = get_tfa();
           </div>
           <div class="form-group">
             <div class="col-sm-offset-3 col-sm-9">
+              <p class="help-block"><?=$lang['admin']['api_info'];?></p>
               <div class="btn-group">
                 <button class="btn btn-default" name="admin_api" type="submit" href="#"><span class="glyphicon glyphicon-check"></span> <?=$lang['admin']['save'];?></button>
                 <button class="btn btn-info" name="admin_api_regen_key" type="submit" href="#"><?=$lang['admin']['regen_api_key'];?></button>
@@ -113,6 +149,7 @@ $tfa_data = get_tfa();
           </div>
         </form>
         </div>
+
       </div>
     </div>
 
@@ -252,7 +289,7 @@ $tfa_data = get_tfa();
             <form class="form" data-id="transport" role="form" method="post">
               <div class="form-group">
                 <label for="destination"><?=$lang['admin']['destination'];?></label>
-                <input class="form-control input-sm" name="destination" placeholder='example.org, .example.org, *, box@example.org' required>
+                <input class="form-control input-sm" name="destination" placeholder='<?=$lang['admin']['transport_dest_format'];?>' required>
               </div>
               <div class="form-group">
                 <label for="nexthop"><?=$lang['admin']['nexthop'];?></label>
@@ -266,6 +303,16 @@ $tfa_data = get_tfa();
                 <label for="password"><?=$lang['admin']['password'];?></label>
                 <input class="form-control" name="password">
               </div>
+              <!-- <div class="form-group">
+                <label>
+                  <input type="checkbox" name="lookup_mx" value="1"> <?=$lang['admin']['lookup_mx'];?>
+                </label>
+              </div> -->
+              <div class="form-group">
+                <label>
+                  <input type="checkbox" name="active" value="1"> <?=$lang['admin']['active'];?>
+                </label>
+              </div>
               <p class="help-block"><?=$lang['admin']['credentials_transport_warning'];?></p>
               <button class="btn btn-default" data-action="add_item" data-id="transport" data-api-url='add/transport' data-api-attr='{}' href="#"><span class="glyphicon glyphicon-plus"></span> <?=$lang['admin']['add'];?></button>
             </form>
@@ -326,7 +373,7 @@ $tfa_data = get_tfa();
           else {
           ?>
           <div class="row">
-              <div class="col-md-1"><input class="dkim_missing" type="checkbox" data-id="dkim" name="multi_select" value="<?=$domain;?>" disabled /></div>
+            <div class="col-md-1"><input class="dkim_missing" type="checkbox" data-id="dkim" name="multi_select" value="<?=$domain;?>" disabled /></div>
             <div class="col-md-3">
               <p><?=$lang['admin']['domain'];?>: <strong><?=htmlspecialchars($domain);?></strong><br /><span class="label label-danger"><?=$lang['admin']['dkim_key_missing'];?></span></p>
             </div>
@@ -600,13 +647,14 @@ $tfa_data = get_tfa();
           </span></p>
           <?php
           endforeach;
+          ?>
+          <hr>
+          <?php
         endif;
         if (!empty($f2b_data['perm_bans'])):
           foreach ($f2b_data['perm_bans'] as $perm_bans):
           ?>
-          <p>
+          <span class="label label-danger" style="padding: 0.1em 0.4em 0.1em;"><span class="glyphicon glyphicon-filter"></span> <?=$perm_bans?></span>
-          <span class="label label-danger" style="padding:4px;font-size:85%;"><span class="glyphicon glyphicon-filter"></span> <?=$perm_bans?></span>
-          </p>
           <?php
           endforeach;
         endif;
@@ -621,30 +669,36 @@ $tfa_data = get_tfa();
        <?php $q_data = quarantine('settings');?>
         <form class="form" data-id="quarantine" role="form" method="post">
           <div class="row">
-            <div class="col-sm-6">
+            <div class="col-sm-4">
               <div class="form-group">
                 <label for="retention_size"><?=$lang['admin']['quarantine_retention_size'];?></label>
                 <input type="number" class="form-control" name="retention_size" value="<?=$q_data['retention_size'];?>" placeholder="0" required>
               </div>
             </div>
-            <div class="col-sm-6">
+            <div class="col-sm-4">
               <div class="form-group">
                 <label for="max_size"><?=$lang['admin']['quarantine_max_size'];?></label>
                 <input type="number" class="form-control" name="max_size" value="<?=$q_data['max_size'];?>" placeholder="0" required>
               </div>
             </div>
+            <div class="col-sm-4">
+              <div class="form-group">
+                <label for="max_age"><?=$lang['admin']['quarantine_max_age'];?></label>
+                <input type="number" class="form-control" name="max_age" value="<?=$q_data['max_age'];?>" min="1" required>
+              </div>
+            </div>
           </div>
           <div class="row">
             <div class="col-sm-6">
               <div class="form-group">
                 <label for="sender"><?=$lang['admin']['quarantine_notification_sender'];?>:</label>
-                <input type="text" class="form-control" name="sender" value="<?=$q_data['sender'];?>" placeholder="quarantine@localhost">
+                <input type="text" class="form-control" name="sender" value="<?=htmlspecialchars($q_data['sender']);?>" placeholder="quarantine@localhost">
               </div>
             </div>
             <div class="col-sm-6">
               <div class="form-group">
                 <label for="subject"><?=$lang['admin']['quarantine_notification_subject'];?>:</label>
-                <input type="text" class="form-control" name="subject" value="<?=$q_data['subject'];?>" placeholder="Spam Quarantine Notification">
+                <input type="text" class="form-control" name="subject" value="<?=htmlspecialchars($q_data['subject']);?>" placeholder="Spam Quarantine Notification">
               </div>
             </div>
           </div>
@@ -699,13 +753,13 @@ $tfa_data = get_tfa();
           <div class="col-sm-6">
             <div class="form-group">
               <label for="sender"><?=$lang['admin']['quarantine_notification_sender'];?>:</label>
-              <input type="text" class="form-control" name="sender" value="<?=$qw_data['sender'];?>" placeholder="quota-warning@localhost">
+              <input type="text" class="form-control" name="sender" value="<?=htmlspecialchars($qw_data['sender']);?>" placeholder="quota-warning@localhost">
             </div>
           </div>
           <div class="col-sm-6">
             <div class="form-group">
               <label for="subject"><?=$lang['admin']['quarantine_notification_subject'];?>:</label>
-              <input type="text" class="form-control" name="subject" value="<?=$qw_data['subject'];?>" placeholder="Quota warning">
+              <input type="text" class="form-control" name="subject" value="<?=htmlspecialchars($qw_data['subject']);?>" placeholder="Quota warning">
             </div>
           </div>
         </div>
@@ -746,6 +800,7 @@ $tfa_data = get_tfa();
       <div id="active_settings_map" class="collapse" >
         <textarea autocorrect="off" spellcheck="false" autocapitalize="none" class="form-control textarea-code" rows="20" name="settings_map" readonly><?=file_get_contents('http://nginx:8081/settings.php');?></textarea>
       </div>
+      <br>
       <?php $rsettings = rsettings('get'); ?>
         <form class="form" data-id="rsettings" role="form" method="post">
           <div class="row">
@@ -796,11 +851,11 @@ $tfa_data = get_tfa();
                     <input type="hidden" name="active" value="0">
                     <div class="form-group">
                       <label for="desc"><?=$lang['admin']['rsetting_desc'];?>:</label>
-                      <input type="text" class="form-control" name="desc" value="<?=$rsetting_details['desc'];?>">
+                      <input type="text" class="form-control" name="desc" value="<?=htmlspecialchars($rsetting_details['desc']);?>">
                     </div>
                     <div class="form-group">
                       <label for="content"><?=$lang['admin']['rsetting_content'];?>:</label>
-                      <textarea class="form-control" name="content" rows="10"><?=$rsetting_details['content'];?></textarea>
+                      <textarea class="form-control" name="content" rows="10"><?=htmlspecialchars($rsetting_details['content']);?></textarea>
                     </div>
                     <div class="form-group">
                       <label>

data/web/css/build/004-bootstrap-slider.min.css

@@ -1,5 +1,5 @@
 /*! =======================================================
-                      VERSION  10.6.0              
+                      VERSION  10.6.1              
 ========================================================= */
 /*! =========================================================
  * bootstrap-slider.js

data/web/css/build/008-mailcow.css

@@ -42,6 +42,9 @@
 .btn {
   text-transform: none;
 }
+.btn * {
+  pointer-events: none;
+}
 .textarea-code {
   font-family:Consolas,Monaco,Lucida Console,Liberation Mono,DejaVu Sans Mono,Bitstream Vera Sans Mono,Courier New, monospace;
   background:transparent !important;

data/web/css/site/admin.css

@@ -71,3 +71,9 @@ body.modal-open {
 .table-condensed > thead > tr > th, .table-condensed > tbody > tr > th, .table-condensed > tfoot > tr > th, .table-condensed > thead > tr > td, .table-condensed > tbody > tr > td, .table-condensed > tfoot > tr > td {
   padding: 3px;
 }
+table tbody tr {
+  cursor: pointer;
+}
+table tbody tr td input[type="checkbox"] {
+  cursor: pointer;
+}

data/web/css/site/index.css

@@ -0,0 +1,5 @@
+@media (max-width: 500px) {
+  #top {
+    padding-top: 15px !important;
+  }
+}

data/web/css/site/mailbox.css

@@ -9,7 +9,7 @@ table.footable>tbody>tr.footable-empty>td {
   overflow: visible !important;
 }
 .table-responsive {
-  overflow: auto !important;
+  overflow: inherit !important;
 }
 @media screen and (max-width: 767px) {
   .table-responsive {
@@ -53,3 +53,9 @@ table.footable>tbody>tr.footable-empty>td {
   font-family:Consolas,Monaco,Lucida Console,Liberation Mono,DejaVu Sans Mono,Bitstream Vera Sans Mono,Courier New, monospace;
   font-size:smaller;
 }
+table tbody tr {
+  cursor: pointer;
+}
+table tbody tr td input[type="checkbox"] {
+  cursor: pointer;
+}

data/web/css/site/quarantine.css

@@ -49,3 +49,19 @@ table.footable>tbody>tr.footable-empty>td {
   border-radius: 50%;
   display: inline-block;
 }
+
+span.mail-address-item {
+  background-color: #f5f5f5;
+  border-radius: 4px;
+  border: 1px solid #ccc;
+  padding: 2px 7px;
+  margin-right: 7px;
+}
+
+table tbody tr {
+  cursor: pointer;
+}
+
+table tbody tr td input[type="checkbox"] {
+  cursor: pointer;
+}

data/web/css/site/user.css

@@ -40,3 +40,11 @@ table.footable>tbody>tr.footable-empty>td {
 body {
   overflow-y:scroll;
 }
+
+table tbody tr {
+  cursor: pointer;
+}
+
+table tbody tr td input[type="checkbox"] {
+  cursor: pointer;
+}

data/web/edit.php

@@ -273,6 +273,12 @@ if (isset($_SESSION['mailcow_cc_role'])) {
                 <input type="number" class="form-control" name="mailboxes" value="<?=intval($result['max_num_mboxes_for_domain']);?>">
               </div>
             </div>
+            <div class="form-group">
+                <label class="control-label col-sm-2" for="defquota"><?=$lang['edit']['mailbox_quota_def'];?></label>
+                <div class="col-sm-10">
+                    <input type="number" class="form-control" name="defquota" value="<?=intval($result['def_quota_for_mbox'] / 1048576);?>">
+                </div>
+            </div>
             <div class="form-group">
               <label class="control-label col-sm-2" for="maxquota"><?=$lang['edit']['max_quota'];?></label>
               <div class="col-sm-10">
@@ -379,7 +385,6 @@ if (isset($_SESSION['mailcow_cc_role'])) {
             <div class="btn-group" data-acl="<?=$_SESSION['acl']['spam_policy'];?>">
               <a class="btn btn-sm btn-default" id="toggle_multi_select_all" data-id="policy_wl_domain" href="#"><span class="glyphicon glyphicon-check" aria-hidden="true"></span> <?=$lang['mailbox']['toggle_all'];?></a>
               <a class="btn btn-sm btn-danger" data-action="delete_selected" data-id="policy_wl_domain" data-api-url='delete/domain-policy' href="#"><?=$lang['mailbox']['remove'];?></a></li>
-              </ul>
             </div>
           </div>
           <form class="form-inline" data-id="add_wl_policy_domain">
@@ -401,7 +406,6 @@ if (isset($_SESSION['mailcow_cc_role'])) {
             <div class="btn-group" data-acl="<?=$_SESSION['acl']['spam_policy'];?>">
               <a class="btn btn-sm btn-default" id="toggle_multi_select_all" data-id="policy_bl_domain" href="#"><span class="glyphicon glyphicon-check" aria-hidden="true"></span> <?=$lang['mailbox']['toggle_all'];?></a>
               <a class="btn btn-sm btn-danger" data-action="delete_selected" data-id="policy_bl_domain" data-api-url='delete/domain-policy' href="#"><?=$lang['mailbox']['remove'];?></a></li>
-              </ul>
             </div>
           </div>
           <form class="form-inline" data-id="add_bl_policy_domain">
@@ -502,6 +506,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
       $mailbox = html_entity_decode(rawurldecode($_GET["mailbox"]));
       $result = mailbox('get', 'mailbox_details', $mailbox);
       $rl = ratelimit('get', 'mailbox', $mailbox);
+      $quarantine_notification = mailbox('get', 'quarantine_notification', $mailbox);
       if (!empty($result)) {
         ?>
         <h4><?=$lang['edit']['mailbox'];?></h4>
@@ -511,21 +516,22 @@ if (isset($_SESSION['mailcow_cc_role'])) {
           <input type="hidden" value="0" name="force_pw_update">
           <input type="hidden" value="0" name="sogo_access">
           <div class="form-group">
-            <label class="control-label col-sm-2" for="name"><?=$lang['edit']['full_name'];?>:</label>
+            <label class="control-label col-sm-2" for="name"><?=$lang['edit']['full_name'];?></label>
             <div class="col-sm-10">
             <input type="text" class="form-control" name="name" value="<?=htmlspecialchars($result['name'], ENT_QUOTES, 'UTF-8');?>">
             </div>
           </div>
           <div class="form-group">
-            <label class="control-label col-sm-2" for="quota"><?=$lang['edit']['quota_mb'];?>:
+            <label class="control-label col-sm-2" for="quota"><?=$lang['edit']['quota_mb'];?>
               <br /><span id="quotaBadge" class="badge">max. <?=intval($result['max_new_quota'] / 1048576)?> MiB</span>
             </label>
             <div class="col-sm-10">
-              <input type="number" name="quota" style="width:100%" min="1" max="<?=intval($result['max_new_quota'] / 1048576);?>" value="<?=intval($result['quota']) / 1048576;?>" class="form-control">
+              <input type="number" name="quota" style="width:100%" min="0" max="<?=intval($result['max_new_quota'] / 1048576);?>" value="<?=intval($result['quota']) / 1048576;?>" class="form-control">
+              <small class="help-block">0 = ∞</small>
             </div>
           </div>
           <div class="form-group">
-            <label class="control-label col-sm-2" for="sender_acl"><?=$lang['edit']['sender_acl'];?>:</label>
+            <label class="control-label col-sm-2" for="sender_acl"><?=$lang['edit']['sender_acl'];?></label>
             <div class="col-sm-10">
               <select data-live-search="true" data-width="100%" style="width:100%" id="editSelectSenderACL" name="sender_acl" size="10" multiple>
               <?php
@@ -537,7 +543,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
                 <?php
               endforeach;
 
-              foreach ($sender_acl_handles['sender_acl_addresses']['ro'] as $domain):
+              foreach ($sender_acl_handles['sender_acl_addresses']['ro'] as $alias):
                 ?>
               <option data-subtext="Admin" disabled selected><?=htmlspecialchars($alias);?></option>
                 <?php
@@ -573,9 +579,51 @@ if (isset($_SESSION['mailcow_cc_role'])) {
                 <?php
               endforeach;
 
+              // Generated here, but used in extended_sender_acl
+              if (!empty($sender_acl_handles['external_sender_aliases'])) {
+                $ext_sender_acl = implode(', ', $sender_acl_handles['external_sender_aliases']);
+              }
+              else {
+                $ext_sender_acl = '';
+              }
+
               ?>
               </select>
               <div style="display:none" id="sender_acl_disabled"><?=$lang['edit']['sender_acl_disabled'];?></div>
+              <small class="help-block"><?=$lang['edit']['sender_acl_info'];?></small>
+            </div>
+          </div>
+          <div class="form-group">
+            <label class="control-label col-sm-2" for="sender_acl"><?=$lang['user']['quarantine_notification'];?></label>
+            <div class="col-sm-10">
+            <div class="btn-group" data-acl="<?=$_SESSION['acl']['quarantine_notification'];?>">
+              <button type="button" class="btn btn-sm btn-default <?=($quarantine_notification == "never") ? "active" : null;?>"
+                data-action="edit_selected"
+                data-item="<?= htmlentities($mailbox); ?>"
+                data-id="quarantine_notification"
+                data-api-url='edit/quarantine_notification'
+                data-api-attr='{"quarantine_notification":"never"}'><?=$lang['user']['never'];?></button>
+              <button type="button" class="btn btn-sm btn-default <?=($quarantine_notification == "hourly") ? "active" : null;?>"
+                data-action="edit_selected"
+                data-item="<?= htmlentities($mailbox); ?>"
+                data-id="quarantine_notification"
+                data-api-url='edit/quarantine_notification'
+                data-api-attr='{"quarantine_notification":"hourly"}'><?=$lang['user']['hourly'];?></button>
+              <button type="button" class="btn btn-sm btn-default <?=($quarantine_notification == "daily") ? "active" : null;?>"
+                data-action="edit_selected"
+                data-item="<?= htmlentities($mailbox); ?>"
+                data-id="quarantine_notification"
+                data-api-url='edit/quarantine_notification'
+                data-api-attr='{"quarantine_notification":"daily"}'><?=$lang['user']['daily'];?></button>
+              <button type="button" class="btn btn-sm btn-default <?=($quarantine_notification == "weekly") ? "active" : null;?>"
+                data-action="edit_selected"
+                data-item="<?= htmlentities($mailbox); ?>"
+                data-id="quarantine_notification"
+                data-api-url='edit/quarantine_notification'
+                data-api-attr='{"quarantine_notification":"weekly"}'><?=$lang['user']['weekly'];?></button>
+            </div>
+            <div style="display:none" id="user_acl_q_notify_disabled"><?=$lang['edit']['user_acl_q_notify_disabled'];?></div>
+            <p class="help-block"><small><?=$lang['user']['quarantine_notification_info'];?></small></p>
             </div>
           </div>
           <div class="form-group">
@@ -590,6 +638,13 @@ if (isset($_SESSION['mailcow_cc_role'])) {
             <input type="password" class="form-control" name="password2">
             </div>
           </div>
+          <div data-acl="<?=$_SESSION['acl']['extend_sender_acl'];?>" class="form-group">
+            <label class="control-label col-sm-2" for="extended_sender_acl"><?=$lang['edit']['extended_sender_acl'];?></label>
+            <div class="col-sm-10">
+            <input type="text" class="form-control" name="extended_sender_acl" value="<?=empty($ext_sender_acl) ? '' : $ext_sender_acl; ?>" placeholder="user1@example.com, user2@example.org, @example.com, ...">
+            <small class="help-block"><?=$lang['edit']['extended_sender_acl_info'];?></small>
+            </div>
+          </div>
           <div class="form-group">
             <div class="col-sm-offset-2 col-sm-10">
               <div class="checkbox">
@@ -639,6 +694,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
               <div class="form-group">
                 <button class="btn btn-default" data-action="edit_selected" data-id="mboxratelimit" data-item="<?=htmlspecialchars($mailbox);?>" data-api-url='edit/rl-mbox' data-api-attr='{}' href="#"><?=$lang['admin']['save'];?></button>
               </div>
+              <p class="help-block"><?=$lang['edit']['mbox_rl_info'];?></p>
             </div>
           </div>
         </form>
@@ -681,6 +737,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
               <label class="control-label col-sm-2" for="hostname"><?=$lang['add']['hostname'];?></label>
               <div class="col-sm-10">
                 <input type="text" class="form-control" name="hostname" value="<?=htmlspecialchars($result['hostname'], ENT_QUOTES, 'UTF-8');?>" required>
+                <p class="help-block"><?=$lang['add']['relayhost_wrapped_tls_info'];?></p>
               </div>
             </div>
             <div class="form-group">
@@ -784,7 +841,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
               </div>
             </div>
             <div class="form-group">
-              <label class="control-label col-sm-2" for="domain"><?=$lang['edit']['kind'];?>:</label>
+              <label class="control-label col-sm-2" for="domain"><?=$lang['edit']['kind'];?></label>
               <div class="col-sm-10">
                 <select name="kind" title="<?=$lang['edit']['select'];?>" required>
                   <option value="location" <?=($result['kind'] == "location") ? "selected" : null;?>>Location</option>
@@ -794,7 +851,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
               </div>
             </div>
             <div class="form-group">
-              <label class="control-label col-sm-2" for="multiple_bookings_select"><?=$lang['add']['multiple_bookings'];?>:</label>
+              <label class="control-label col-sm-2" for="multiple_bookings_select"><?=$lang['add']['multiple_bookings'];?></label>
               <div class="col-sm-10">
                 <select name="multiple_bookings_select" id="editSelectMultipleBookings" title="<?=$lang['add']['select'];?>" required>
                   <option value="0" <?=($result['multiple_bookings'] == 0) ? "selected" : null;?>><?=$lang['mailbox']['booking_0'];?></option>
@@ -1027,7 +1084,7 @@ if (isset($_SESSION['mailcow_cc_role'])) {
               </div>
             </div>
             <div class="form-group">
-              <label class="control-label col-sm-2" for="enc1"><?=$lang['edit']['encryption'];?>:</label>
+              <label class="control-label col-sm-2" for="enc1"><?=$lang['edit']['encryption'];?></label>
               <div class="col-sm-10">
                 <select id="enc1" name="enc1">
                   <option <?=($result['enc1'] == "TLS") ? "selected" : null;?>>TLS</option>
@@ -1086,7 +1143,8 @@ if (isset($_SESSION['mailcow_cc_role'])) {
             <div class="form-group">
               <label class="control-label col-sm-2" for="custom_params"><?=$lang['add']['custom_params'];?></label>
               <div class="col-sm-10">
-              <input type="text" class="form-control" name="custom_params" id="custom_params" value="<?=htmlspecialchars($result['custom_params'], ENT_QUOTES, 'UTF-8');?>">
+              <input type="text" class="form-control" name="custom_params" id="custom_params" value="<?=htmlspecialchars($result['custom_params'], ENT_QUOTES, 'UTF-8');?>" placeholder="--dry --some-param=xy --other-param=yx">
+              <small class="help-block"><?=$lang['add']['custom_params_hint'];?></small>
               </div>
             </div>
             <div class="form-group">
@@ -1162,13 +1220,13 @@ if (isset($_SESSION['mailcow_cc_role'])) {
           <form class="form-horizontal" data-id="editfilter" role="form" method="post">
             <input type="hidden" value="0" name="active">
             <div class="form-group">
-              <label class="control-label col-sm-2" for="script_desc"><?=$lang['edit']['sieve_desc'];?>:</label>
+              <label class="control-label col-sm-2" for="script_desc"><?=$lang['edit']['sieve_desc'];?></label>
               <div class="col-sm-10">
               <input type="text" class="form-control" name="script_desc" id="script_desc" value="<?=htmlspecialchars($result['script_desc'], ENT_QUOTES, 'UTF-8');?>" required maxlength="255">
               </div>
             </div>
             <div class="form-group">
-              <label class="control-label col-sm-2" for="filter_type"><?=$lang['edit']['sieve_type'];?>:</label>
+              <label class="control-label col-sm-2" for="filter_type"><?=$lang['edit']['sieve_type'];?></label>
               <div class="col-sm-10">
                 <select id="addFilterType" name="filter_type" id="filter_type" required>
                   <option value="prefilter" <?=($result['filter_type'] == 'prefilter') ? 'selected' : null;?>>Prefilter</option>

data/web/inc/ajax/dns_diagnostics.php

@@ -75,7 +75,7 @@ if (!isset($autodiscover_config['sieve'])) {
 }
 
 // Init records array
-$spf_link = '<a href="http://www.openspf.org/SPF_Record_Syntax" target="_blank">SPF Record Syntax</a><br />';
+$spf_link = '<a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework" target="_blank">SPF Record Syntax</a><br />';
 $dmarc_link = '<a href="https://www.kitterman.com/dmarc/assistant.html" target="_blank">DMARC Assistant</a>';
 
 $records = array();

data/web/inc/ajax/qitem_details.php

@@ -3,8 +3,9 @@ session_start();
 header("Content-Type: application/json");
 require_once $_SERVER['DOCUMENT_ROOT'] . '/inc/prerequisites.inc.php';
 if (!isset($_SESSION['mailcow_cc_role'])) {
-	exit();
+  exit();
 }
+
 function rrmdir($src) {
   $dir = opendir($src);
   while(false !== ( $file = readdir($dir)) ) {
@@ -21,6 +22,13 @@ function rrmdir($src) {
   closedir($dir);
   rmdir($src);
 }
+function addAddresses(&$list, $mail, $headerName) {
+  $addresses = $mail->getAddresses($headerName);
+  foreach ($addresses as $address) {
+    $list[] = array('address' => $address['address'], 'type' => $headerName);
+  }
+}
+
 if (!empty($_GET['id']) && ctype_alnum($_GET['id'])) {
   $tmpdir = '/tmp/' . $_GET['id'] . '/';
   $mailc = quarantine('details', $_GET['id']);
@@ -36,6 +44,16 @@ if (!empty($_GET['id']) && ctype_alnum($_GET['id'])) {
     $html2text = new Html2Text\Html2Text();
     // Load msg to parser
     $mail_parser->setText($mailc['msg']);
+
+    // Get mail recipients
+    {
+      $recipientsList = array();
+      addAddresses($recipientsList, $mail_parser, 'to');
+      addAddresses($recipientsList, $mail_parser, 'cc');
+      addAddresses($recipientsList, $mail_parser, 'bcc');
+      $data['recipients'] = $recipientsList;
+    }
+
     // Get text/plain content
     $data['text_plain'] = $mail_parser->getMessageBody('text');
     // Get html content and convert to text

data/web/inc/ajax/qr_gen.php

@@ -0,0 +1,13 @@
+<?php
+session_start();
+require_once $_SERVER['DOCUMENT_ROOT'] . '/inc/prerequisites.inc.php';
+header('Content-Type: text/plain');
+if (!isset($_SESSION['mailcow_cc_role'])) {
+	exit();
+}
+
+if (isset($_GET['token']) && ctype_alnum($_GET['token'])) {
+  echo $tfa->getQRCodeImageAsDataUri($_SESSION['mailcow_cc_username'], $_GET['token']);
+}
+
+?>

data/web/inc/ajax/transport_check.php

@@ -58,6 +58,11 @@ if (isset($_SESSION['mailcow_cc_role']) && $_SESSION['mailcow_cc_role'] == "admi
       )
     );
     $mail->SMTPDebug = 3;
+    // smtp: and smtp_enforced_tls: do not support wrapped tls, todo?
+    // change postfix map to detect wrapped tls or add a checkbox to toggle wrapped tls
+    // if ($port == 465) {
+      // $mail->SMTPSecure = "ssl";
+    // }
     $mail->Debugoutput = function($str, $level) {
       foreach(preg_split("/((\r?\n)|(\r\n?)|\n)/", $str) as $line){
         if (empty($line)) { continue; }

data/web/inc/footer.inc.php

@@ -26,6 +26,10 @@ $(window).load(function() {
   $(".overlay").hide();
 });
 $(document).ready(function() {
+  $(document).on('shown.bs.modal', function(e) {
+    modal_id = $(e.relatedTarget).data('target');
+    $(modal_id).attr("aria-hidden","false");
+  });
   // TFA, CSRF, Alerts in footer.inc.php
   // Other general functions in mailcow.js
   <?php
@@ -93,6 +97,15 @@ $(document).ready(function() {
     }
     if ($(this).val() == "totp") {
       $('#TOTPModal').modal('show');
+      request_token = $('#tfa-qr-img').data('totp-secret');
+      $.ajax({
+        url: '/inc/ajax/qr_gen.php',
+        data: {
+          token: request_token,
+        },
+      }).done(function (result) {
+        $("#tfa-qr-img").attr("src", result);
+      });
       $("option:selected").prop("selected", false);
     }
     if ($(this).val() == "u2f") {

data/web/inc/functions.address_rewriting.inc.php

@@ -69,7 +69,7 @@ function bcc($_action, $_data = null, $attr = null) {
         $_SESSION['return'][] = array(
           'type' => 'danger',
           'log' => array(__FUNCTION__, $_action, $_data, $_attr),
-          'msg' => 'bcc_must_be_email'
+          'msg' => array('bcc_must_be_email', htmlspecialchars($bcc_dest))
         );
         return false;
       }

data/web/inc/functions.fail2ban.inc.php

@@ -9,6 +9,11 @@ function valid_network($network) {
   }
   return false;
 }
+
+function valid_hostname($hostname) {
+    return filter_var($hostname, FILTER_VALIDATE_DOMAIN, FILTER_FLAG_HOSTNAME);
+}
+
 function fail2ban($_action, $_data = null) {
   global $redis;
   global $lang;
@@ -188,7 +193,7 @@ function fail2ban($_action, $_data = null) {
           $wl_array = array_map('trim', preg_split( "/( |,|;|\n)/", $wl));
           if (is_array($wl_array)) {
             foreach ($wl_array as $wl_item) {
-              if (valid_network($wl_item)) {
+              if (valid_network($wl_item) || valid_hostname($wl_item)) {
                 $redis->hSet('F2B_WHITELIST', $wl_item, 1);
               }
             }
@@ -198,7 +203,7 @@ function fail2ban($_action, $_data = null) {
           $bl_array = array_map('trim', preg_split( "/( |,|;|\n)/", $bl));
           if (is_array($bl_array)) {
             foreach ($bl_array as $bl_item) {
-              if (valid_network($bl_item)) {
+              if (valid_network($bl_item) || valid_hostname($bl_item)) {
                 $redis->hSet('F2B_BLACKLIST', $bl_item, 1);
               }
             }

data/web/inc/functions.inc.php

@@ -1,4 +1,12 @@
 <?php
+function isset_has_content($var) {
+  if (isset($var) && $var != "") {
+    return true;
+  }
+  else {
+    return false;
+  }
+}
 function hash_password($password) {
 	$salt_str = bin2hex(openssl_random_pseudo_bytes(8));
 	return "{SSHA256}".base64_encode(hash('sha256', $password . $salt_str, true) . $salt_str);
@@ -248,6 +256,25 @@ function hasMailboxObjectAccess($username, $role, $object) {
   }
 	return false;
 }
+function hasAliasObjectAccess($username, $role, $object) {
+	global $pdo;
+	if (!filter_var(html_entity_decode(rawurldecode($username)), FILTER_VALIDATE_EMAIL) && !ctype_alnum(str_replace(array('_', '.', '-'), '', $username))) {
+		return false;
+	}
+	if ($role != 'admin' && $role != 'domainadmin' && $role != 'user') {
+		return false;
+	}
+	if ($username == $object) {
+		return true;
+	}
+  $stmt = $pdo->prepare("SELECT `domain` FROM `alias` WHERE `address` = :object");
+  $stmt->execute(array(':object' => $object));
+  $row = $stmt->fetch(PDO::FETCH_ASSOC);
+  if (isset($row['domain']) && hasDomainAccess($username, $role, $row['domain'])) {
+    return true;
+  }
+	return false;
+}
 function pem_to_der($pem_key) {
   // Need to remove BEGIN/END PUBLIC KEY
   $lines = explode("\n", trim($pem_key));
@@ -525,8 +552,8 @@ function update_sogo_static_view() {
     WHERE TABLE_NAME = 'sogo_view'");
   $num_results = count($stmt->fetchAll(PDO::FETCH_ASSOC));
   if ($num_results != 0) {
-    $stmt = $pdo->query("REPLACE INTO _sogo_static_view (`c_uid`, `domain`, `c_name`, `c_password`, `c_cn`, `mail`, `aliases`, `ad_aliases`, `kind`, `multiple_bookings`)
+    $stmt = $pdo->query("REPLACE INTO _sogo_static_view (`c_uid`, `domain`, `c_name`, `c_password`, `c_cn`, `mail`, `aliases`, `ad_aliases`, `ext_acl`, `kind`, `multiple_bookings`)
-      SELECT `c_uid`, `domain`, `c_name`, `c_password`, `c_cn`, `mail`, `aliases`, `ad_aliases`, `kind`, `multiple_bookings` from sogo_view");
+      SELECT `c_uid`, `domain`, `c_name`, `c_password`, `c_cn`, `mail`, `aliases`, `ad_aliases`, `ext_acl`, `kind`, `multiple_bookings` from sogo_view");
     $stmt = $pdo->query("DELETE FROM _sogo_static_view WHERE `c_uid` NOT IN (SELECT `username` FROM `mailbox` WHERE `active` = '1');");
   }
   flush_memcached();
@@ -668,7 +695,7 @@ function user_get_alias_details($username) {
   while ($row = array_shift($run)) {
     $data['aliases_also_send_as'] = $row['send_as'];
   }
-  $stmt = $pdo->prepare("SELECT IFNULL(CONCAT(GROUP_CONCAT(DISTINCT `send_as` SEPARATOR ', '), ', ', GROUP_CONCAT(DISTINCT CONCAT('@',`alias_domain`) SEPARATOR ', ')), '&#10008;') AS `send_as` FROM `sender_acl` LEFT JOIN `alias_domain` ON `alias_domain`.`target_domain` =  TRIM(LEADING '@' FROM `send_as`) WHERE `logged_in_as` = :username AND `send_as` LIKE '@%';");
+  $stmt = $pdo->prepare("SELECT CONCAT_WS(', ', IFNULL(GROUP_CONCAT(DISTINCT `send_as` SEPARATOR ', '), '&#10008;'), GROUP_CONCAT(DISTINCT CONCAT('@',`alias_domain`) SEPARATOR ', ')) AS `send_as` FROM `sender_acl` LEFT JOIN `alias_domain` ON `alias_domain`.`target_domain` =  TRIM(LEADING '@' FROM `send_as`) WHERE `logged_in_as` = :username AND `send_as` LIKE '@%';");
   $stmt->execute(array(':username' => $username));
   $run = $stmt->fetchAll(PDO::FETCH_ASSOC);
   while ($row = array_shift($run)) {
@@ -1196,6 +1223,69 @@ function admin_api($action, $data = null) {
 		'msg' => 'admin_api_modified'
 	);
 }
+function license($action, $data = null) {
+	global $pdo;
+	global $redis;
+	global $lang;
+	if ($_SESSION['mailcow_cc_role'] != "admin") {
+		$_SESSION['return'][] =  array(
+			'type' => 'danger',
+      'log' => array(__FUNCTION__),
+			'msg' => 'access_denied'
+		);
+		return false;
+	}
+	switch ($action) {
+		case "verify":
+      // Keep result until revalidate button is pressed or session expired
+      $stmt = $pdo->query("SELECT `version` FROM `versions` WHERE `application` = 'GUID'");
+      $versions = $stmt->fetch(PDO::FETCH_ASSOC);
+      $post = array('guid' => $versions['version']);
+      $curl = curl_init('https://verify.mailcow.email');
+      curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
+      curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
+      curl_setopt($curl, CURLOPT_POSTFIELDS, $post);
+      $response = curl_exec($curl);
+      curl_close($curl);
+      $json_return = json_decode($response, true);
+      if ($response && $json_return) {
+        if ($json_return['response'] === "ok") {
+          $_SESSION['gal']['valid'] = "true";
+          $_SESSION['gal']['c'] = $json_return['c'];
+          $_SESSION['gal']['s'] = $json_return['s'];
+                  }
+        elseif ($json_return['response'] === "invalid") {
+          $_SESSION['gal']['valid'] = "false";
+          $_SESSION['gal']['c'] = $lang['mailbox']['no'];
+          $_SESSION['gal']['s'] = $lang['mailbox']['no'];
+        }
+      }
+      else {
+        $_SESSION['gal']['valid'] = "false";
+        $_SESSION['gal']['c'] = $lang['danger']['temp_error'];
+        $_SESSION['gal']['s'] = $lang['danger']['temp_error'];
+      }
+      try {
+        // json_encode needs "true"/"false" instead of true/false, to not encode it to 0 or 1
+        $redis->Set('LICENSE_STATUS_CACHE', json_encode($_SESSION['gal']));
+      }
+      catch (RedisException $e) {
+        $_SESSION['return'][] = array(
+          'type' => 'danger',
+          'log' => array(__FUNCTION__, $_action, $_data_log),
+          'msg' => array('redis_error', $e)
+        );
+        return false;
+      }
+      return $_SESSION['gal']['valid'];
+    break;
+    case "guid":
+      $stmt = $pdo->query("SELECT `version` FROM `versions` WHERE `application` = 'GUID'");
+      $versions = $stmt->fetch(PDO::FETCH_ASSOC);
+      return $versions['version'];
+    break;
+  }
+}
 function rspamd_ui($action, $data = null) {
 	global $lang;
 	if ($_SESSION['mailcow_cc_role'] != "admin") {
@@ -1477,7 +1567,7 @@ function solr_status() {
   $endpoint = 'http://solr:8983/solr/admin/cores';
   $params = array(
     'action' => 'STATUS',
-    'core' => 'dovecot',
+    'core' => 'dovecot-fts',
     'indexInfo' => 'true'
   );
   $url = $endpoint . '?' . http_build_query($params);
@@ -1494,7 +1584,7 @@ function solr_status() {
   else {
     curl_close($curl);
     $status = json_decode($response, true);
-    return (!empty($status['status']['dovecot'])) ? $status['status']['dovecot'] : false;
+    return (!empty($status['status']['dovecot-fts'])) ? $status['status']['dovecot-fts'] : false;
   }
   return false;
 }

data/web/inc/functions.mailbox.inc.php

@@ -326,9 +326,18 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           $description  = $_data['description'];
           $aliases			= $_data['aliases'];
           $mailboxes    = $_data['mailboxes'];
+          $defquota			= $_data['defquota'];
           $maxquota			= $_data['maxquota'];
           $restart_sogo = $_data['restart_sogo'];
           $quota				= $_data['quota'];
+          if ($defquota > $maxquota) {
+            $_SESSION['return'][] = array(
+                'type' => 'danger',
+                'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                'msg' => 'mailbox_defquota_exceeds_mailbox_maxquota'
+            );
+            return false;
+          }
           if ($maxquota > $quota) {
             $_SESSION['return'][] = array(
               'type' => 'danger',
@@ -337,6 +346,14 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             );
             return false;
           }
+          if ($defquota == "0" || empty($defquota)) {
+            $_SESSION['return'][] = array(
+                'type' => 'danger',
+                'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                'msg' => 'defquota_empty'
+            );
+            return false;
+          }
           if ($maxquota == "0" || empty($maxquota)) {
             $_SESSION['return'][] = array(
               'type' => 'danger',
@@ -392,13 +409,18 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             );
             return false;
           }
-          $stmt = $pdo->prepare("INSERT INTO `domain` (`domain`, `description`, `aliases`, `mailboxes`, `maxquota`, `quota`, `backupmx`, `gal`, `active`, `relay_all_recipients`)
+          $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `external` = 1 AND `send_as` LIKE :domain");
-            VALUES (:domain, :description, :aliases, :mailboxes, :maxquota, :quota, :backupmx, :gal, :active, :relay_all_recipients)");
+          $stmt->execute(array(
+            ':domain' => '%@' . $domain
+          ));
+          $stmt = $pdo->prepare("INSERT INTO `domain` (`domain`, `description`, `aliases`, `mailboxes`, `defquota`, `maxquota`, `quota`, `backupmx`, `gal`, `active`, `relay_all_recipients`)
+            VALUES (:domain, :description, :aliases, :mailboxes, :defquota, :maxquota, :quota, :backupmx, :gal, :active, :relay_all_recipients)");
           $stmt->execute(array(
             ':domain' => $domain,
             ':description' => $description,
             ':aliases' => $aliases,
             ':mailboxes' => $mailboxes,
+            ':defquota' => $defquota,
             ':maxquota' => $maxquota,
             ':quota' => $quota,
             ':backupmx' => $backupmx,
@@ -561,7 +583,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
                 'msg' => array('is_alias_or_mailbox', htmlspecialchars($address))
               );
-              return false;
+              continue;
             }
             $stmt = $pdo->prepare("SELECT `domain` FROM `domain`
               WHERE `domain`= :domain1 OR `domain` = (SELECT `target_domain` FROM `alias_domain` WHERE `alias_domain` = :domain2)");
@@ -573,7 +595,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
                 'msg' => array('domain_not_found', htmlspecialchars($domain))
               );
-              return false;
+              continue;
             }
             $stmt = $pdo->prepare("SELECT `address` FROM `spamalias`
               WHERE `address`= :address");
@@ -585,7 +607,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
                 'msg' => array('is_spam_alias', htmlspecialchars($address))
               );
-              return false;
+              continue;
             }
             if ((!filter_var($address, FILTER_VALIDATE_EMAIL) === true) && !empty($local_part)) {
               $_SESSION['return'][] = array(
@@ -593,7 +615,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
                 'msg' => 'alias_invalid'
               );
-              return false;
+              continue;
             }
             if (!hasDomainAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $domain)) {
               $_SESSION['return'][] = array(
@@ -601,7 +623,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
                 'msg' => 'access_denied'
               );
-              return false;
+              continue;
             }
             $stmt = $pdo->prepare("INSERT INTO `alias` (`address`, `public_comment`, `private_comment`, `goto`, `domain`, `active`)
               VALUES (:address, :public_comment, :private_comment, :goto, :domain, :active)");
@@ -692,6 +714,18 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               );
               continue;
             }
+            $stmt = $pdo->prepare("SELECT `domain` FROM `domain`
+              WHERE `domain`= :target_domain AND `backupmx` = '1'");
+            $stmt->execute(array(':target_domain' => $target_domain));
+            $num_results = count($stmt->fetchAll(PDO::FETCH_ASSOC));
+            if ($num_results == 1) {
+              $_SESSION['return'][] = array(
+                'type' => 'danger',
+                'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                'msg' => array('targetd_relay_domain', htmlspecialchars($target_domain))
+              );
+              continue;
+            }
             $stmt = $pdo->prepare("SELECT `alias_domain` FROM `alias_domain` WHERE `alias_domain`= :alias_domain
               UNION
               SELECT `domain` FROM `domain` WHERE `domain`= :alias_domain_in_domain");
@@ -705,6 +739,10 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               );
               continue;
             }
+            $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `external` = 1 AND `send_as` LIKE :domain");
+            $stmt->execute(array(
+              ':domain' => '%@' . $domain
+            ));
             $stmt = $pdo->prepare("INSERT INTO `alias_domain` (`alias_domain`, `target_domain`, `active`)
               VALUES (:alias_domain, :target_domain, :active)");
             $stmt->execute(array(
@@ -756,7 +794,15 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           $password     = $_data['password'];
           $password2    = $_data['password2'];
           $name         = ltrim(rtrim($_data['name'], '>'), '<');
-          $quota_m			= filter_var($_data['quota'], FILTER_SANITIZE_NUMBER_FLOAT);
+          $quota_m			= intval($_data['quota']);
+          if ((!isset($_SESSION['acl']['unlimited_quota']) || $_SESSION['acl']['unlimited_quota'] != "1") && $quota_m === 0) {
+            $_SESSION['return'][] = array(
+              'type' => 'danger',
+              'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+              'msg' => 'unlimited_quota_acl'
+            );
+            return false;
+          }
           if (empty($name)) {
             $name = $local_part;
           }
@@ -844,14 +890,6 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             );
             return false;
           }
-          if (!is_numeric($quota_m) || $quota_m == "0") {
-            $_SESSION['return'][] = array(
-              'type' => 'danger',
-              'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
-              'msg' => 'quota_not_0_not_numeric'
-            );
-            return false;
-          }
           if (!empty($password) && !empty($password2)) {
             if (!preg_match('/' . $GLOBALS['PASSWD_REGEP'] . '/', $password)) {
               $_SESSION['return'][] = array(
@@ -1695,25 +1733,27 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 );
                 continue;
               }
-              $stmt = $pdo->prepare("SELECT `address` FROM `alias`
+              if (strtolower($is_now['address']) != strtolower($address)) {
-                WHERE `address`= :address OR `address` IN (
+                $stmt = $pdo->prepare("SELECT `address` FROM `alias`
-                  SELECT `username` FROM `mailbox`, `alias_domain`
+                  WHERE `address`= :address OR `address` IN (
-                    WHERE (
+                    SELECT `username` FROM `mailbox`, `alias_domain`
-                      `alias_domain`.`alias_domain` = :address_d
+                      WHERE (
-                        AND `mailbox`.`username` = CONCAT(:address_l, '@', alias_domain.target_domain)))");
+                        `alias_domain`.`alias_domain` = :address_d
-              $stmt->execute(array(
+                          AND `mailbox`.`username` = CONCAT(:address_l, '@', alias_domain.target_domain)))");
-                ':address' => $address,
+                $stmt->execute(array(
-                ':address_l' => $local_part,
+                  ':address' => $address,
-                ':address_d' => $domain
+                  ':address_l' => $local_part,
-              ));
+                  ':address_d' => $domain
-              $num_results = count($stmt->fetchAll(PDO::FETCH_ASSOC));
+                ));
-              if ($num_results != 0) {
+                $num_results = count($stmt->fetchAll(PDO::FETCH_ASSOC));
-                $_SESSION['return'][] = array(
+                if ($num_results != 0) {
-                  'type' => 'danger',
+                  $_SESSION['return'][] = array(
-                  'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                    'type' => 'danger',
-                  'msg' => array('is_alias_or_mailbox', htmlspecialchars($address))
+                    'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
-                );
+                    'msg' => array('is_alias_or_mailbox', htmlspecialchars($address))
-                continue;
+                  );
+                  continue;
+                }
               }
               $stmt = $pdo->prepare("SELECT `domain` FROM `domain`
                 WHERE `domain`= :domain1 OR `domain` = (SELECT `target_domain` FROM `alias_domain` WHERE `alias_domain` = :domain2)");
@@ -1773,6 +1813,14 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                   unset($gotos[$i]);
                   continue;
                 }
+                // Delete from sender_acl to prevent duplicates
+                $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE
+                  `logged_in_as` = :goto AND
+                  `send_as` = :address");
+                $stmt->execute(array(
+                  ':goto' => $goto,
+                  ':address' => $address
+                ));
               }
               $gotos = array_filter($gotos);
               $goto = implode(",", $gotos);
@@ -1859,6 +1907,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 $relayhost            = (isset($_data['relayhost'])) ? intval($_data['relayhost']) : $is_now['relayhost'];
                 $aliases              = (!empty($_data['aliases'])) ? $_data['aliases'] : $is_now['max_num_aliases_for_domain'];
                 $mailboxes            = (isset($_data['mailboxes']) && $_data['mailboxes'] != '') ? intval($_data['mailboxes']) : $is_now['max_num_mboxes_for_domain'];
+                $defquota             = (isset($_data['defquota']) && $_data['defquota'] != '') ? intval($_data['defquota']) : ($is_now['def_quota_for_mbox'] / 1048576);
                 $maxquota             = (!empty($_data['maxquota'])) ? $_data['maxquota'] : ($is_now['max_quota_for_mbox'] / 1048576);
                 $quota                = (!empty($_data['quota'])) ? $_data['quota'] : ($is_now['max_quota_for_domain'] / 1048576);
                 $description          = (!empty($_data['description'])) ? $_data['description'] : $is_now['description'];
@@ -1890,6 +1939,22 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                   )");
               $stmt->execute(array(':domain' => $domain));
               $AliasData = $stmt->fetch(PDO::FETCH_ASSOC);
+              if ($defquota > $maxquota) {
+                $_SESSION['return'][] = array(
+                    'type' => 'danger',
+                    'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                    'msg' => 'mailbox_defquota_exceeds_mailbox_maxquota'
+                );
+                continue;
+              }
+              if ($defquota == "0" || empty($defquota)) {
+                $_SESSION['return'][] = array(
+                    'type' => 'danger',
+                    'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                    'msg' => 'defquota_empty'
+                );
+                continue;
+              }
               if ($maxquota > $quota) {
                 $_SESSION['return'][] = array(
                   'type' => 'danger',
@@ -1944,6 +2009,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               `gal` = :gal,
               `active` = :active,
               `quota` = :quota,
+              `defquota` = :defquota,
               `maxquota` = :maxquota,
               `relayhost` = :relayhost,
               `mailboxes` = :mailboxes,
@@ -1956,6 +2022,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 ':gal' => $gal,
                 ':active' => $active,
                 ':quota' => $quota,
+                ':defquota' => $defquota,
                 ':maxquota' => $maxquota,
                 ':relayhost' => $relayhost,
                 ':mailboxes' => $mailboxes,
@@ -1993,9 +2060,9 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               $active     = (isset($_data['active'])) ? intval($_data['active']) : $is_now['active_int'];
               (int)$force_pw_update = (isset($_data['force_pw_update'])) ? intval($_data['force_pw_update']) : intval($is_now['attributes']['force_pw_update']);
               (int)$sogo_access = (isset($_data['sogo_access'])) ? intval($_data['sogo_access']) : intval($is_now['attributes']['sogo_access']);
+              (int)$quota_m = (isset_has_content($_data['quota'])) ? intval($_data['quota']) : ($is_now['quota'] / 1048576);
               $name       = (!empty($_data['name'])) ? ltrim(rtrim($_data['name'], '>'), '<') : $is_now['name'];
               $domain     = $is_now['domain'];
-              $quota_m    = (!empty($_data['quota'])) ? $_data['quota'] : ($is_now['quota'] / 1048576);
               $quota_b    = $quota_m * 1048576;
               $password   = (!empty($_data['password'])) ? $_data['password'] : null;
               $password2  = (!empty($_data['password2'])) ? $_data['password2'] : null; 
@@ -2008,6 +2075,15 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               );
               continue;
             }
+            // if already 0 == ok
+            if ((!isset($_SESSION['acl']['unlimited_quota']) || $_SESSION['acl']['unlimited_quota'] != "1") && ($quota_m == 0 && $is_now['quota'] != 0)) {
+              $_SESSION['return'][] = array(
+                'type' => 'danger',
+                'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                'msg' => 'unlimited_quota_acl'
+              );
+              return false;
+            }
             $stmt = $pdo->prepare("SELECT `quota`, `maxquota`
               FROM `domain`
                 WHERE `domain` = :domain");
@@ -2021,14 +2097,6 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               );
               continue;
             }
-            if (!is_numeric($quota_m) || $quota_m == "0") {
-              $_SESSION['return'][] = array(
-                'type' => 'danger',
-                'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
-                'msg' => array('quota_not_0_not_numeric', htmlspecialchars($quota_m))
-              );
-              continue;
-            }
             if ($quota_m > $DomainData['maxquota']) {
               $_SESSION['return'][] = array(
                 'type' => 'danger',
@@ -2045,6 +2113,75 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               );
               continue;
             }
+            $extra_acls = array();
+            if (isset($_data['extended_sender_acl'])) {
+              if (!isset($_SESSION['acl']['extend_sender_acl']) || $_SESSION['acl']['extend_sender_acl'] != "1" ) {
+                $_SESSION['return'][] = array(
+                  'type' => 'danger',
+                  'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                  'msg' => 'access_denied'
+                );
+                return false;
+              }
+              $extra_acls = array_map('trim', preg_split( "/( |,|;|\n)/", $_data['extended_sender_acl']));
+              foreach ($extra_acls as $i => &$extra_acl) {
+                if (empty($extra_acl)) {
+                  continue;
+                }
+                if (substr($extra_acl, 0, 1) === "@") {
+                  $extra_acl = ltrim($extra_acl, '@');
+                }
+                if (!filter_var($extra_acl, FILTER_VALIDATE_EMAIL) && !is_valid_domain_name($extra_acl)) {
+                  $_SESSION['return'][] = array(
+                    'type' => 'danger',
+                    'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                    'msg' => array('extra_acl_invalid', htmlspecialchars($extra_acl))
+                  );
+                  unset($extra_acls[$i]);
+                  continue;
+                }
+                $domains = array_merge(mailbox('get', 'domains'), mailbox('get', 'alias_domains'));
+                if (filter_var($extra_acl, FILTER_VALIDATE_EMAIL)) {
+                  $extra_acl_domain = idn_to_ascii(substr(strstr($extra_acl, '@'), 1), 0, INTL_IDNA_VARIANT_UTS46);
+                  if (in_array($extra_acl_domain, $domains)) {
+                    $_SESSION['return'][] = array(
+                      'type' => 'danger',
+                      'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                      'msg' => array('extra_acl_invalid_domain', $extra_acl_domain)
+                    );
+                    unset($extra_acls[$i]);
+                    continue;
+                  }
+                }
+                else {
+                  if (in_array($extra_acl, $domains)) {
+                    $_SESSION['return'][] = array(
+                      'type' => 'danger',
+                      'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
+                      'msg' => array('extra_acl_invalid_domain', $extra_acl_domain)
+                    );
+                    unset($extra_acls[$i]);
+                    continue;
+                  }
+                  $extra_acl = '@' . $extra_acl;
+                }
+              }
+              $extra_acls = array_filter($extra_acls);
+              $extra_acls = array_values($extra_acls);
+              $extra_acls = array_unique($extra_acls);
+              $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `external` = 1 AND `logged_in_as` = :username");
+              $stmt->execute(array(
+                ':username' => $username
+              ));
+              foreach ($extra_acls as $sender_acl_external) {
+                $stmt = $pdo->prepare("INSERT INTO `sender_acl` (`send_as`, `logged_in_as`, `external`)
+                  VALUES (:sender_acl, :username, 1)");
+                $stmt->execute(array(
+                  ':sender_acl' => $sender_acl_external,
+                  ':username' => $username
+                ));
+              }
+            }
             if (isset($_data['sender_acl'])) {
               // Get sender_acl items set by admin
               $sender_acl_admin = array_merge(
@@ -2116,9 +2253,9 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                     unset($sender_acl_domain_admin[$key]);
                     continue;
                   }
-                  // Check if user has mailbox access (if object is email)
+                  // Check if user has alias access (if object is email)
                   if (filter_var($val, FILTER_VALIDATE_EMAIL)) {
-                    if (!hasMailboxObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $val)) {
+                    if (!hasAliasObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $val)) {
                       $_SESSION['return'][] = array(
                         'type' => 'danger',
                         'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
@@ -2133,15 +2270,20 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 $sender_acl_merged = array_merge($sender_acl_domain_admin, $sender_acl_admin);
                 // If merged array still contains "*", set it as only value
                 !in_array('*', $sender_acl_merged) ?: $sender_acl_merged = array('*');
-                $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `logged_in_as` = :username");
+                $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `external` = 0 AND `logged_in_as` = :username");
                 $stmt->execute(array(
                   ':username' => $username
                 ));
+                $fixed_sender_aliases = mailbox('get', 'sender_acl_handles', $username)['fixed_sender_aliases'];
                 foreach ($sender_acl_merged as $sender_acl) {
                   $domain = ltrim($sender_acl, '@');
                   if (is_valid_domain_name($domain)) {
                     $sender_acl = '@' . $domain;
                   }
+                  // Don't add if allowed by alias
+                  if (in_array($sender_acl, $fixed_sender_aliases)) {
+                    continue;
+                  }
                   $stmt = $pdo->prepare("INSERT INTO `sender_acl` (`send_as`, `logged_in_as`)
                     VALUES (:sender_acl, :username)");
                   $stmt->execute(array(
@@ -2151,7 +2293,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
                 }
               }
               else {
-                $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `logged_in_as` = :username");
+                $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `external` = 0 AND `logged_in_as` = :username");
                 $stmt->execute(array(
                   ':username' => $username
                 ));
@@ -2306,6 +2448,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           $data['sender_acl_addresses']['rw']             = array();
           $data['sender_acl_addresses']['selectable']     = array();
           $data['fixed_sender_aliases']                   = array();
+          $data['external_sender_aliases']                = array();
           // Fixed addresses
           $stmt = $pdo->prepare("SELECT `address` FROM `alias` WHERE `goto` REGEXP :goto AND `address` NOT LIKE '@%'");
           $stmt->execute(array(':goto' => '(^|,)'.$_data.'($|,)'));
@@ -2323,9 +2466,18 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               $data['fixed_sender_aliases'][] = $row['alias_domain_alias'];
             }
           }
+          // External addresses
+          $stmt = $pdo->prepare("SELECT `send_as` as `send_as_external` FROM `sender_acl` WHERE `logged_in_as` = :logged_in_as AND `external` = '1'");
+          $stmt->execute(array(':logged_in_as' => $_data));
+          $exernal_rows = $stmt->fetchAll(PDO::FETCH_ASSOC);
+          while ($row = array_shift($exernal_rows)) {
+            if (!empty($row['send_as_external'])) {
+              $data['external_sender_aliases'][] = $row['send_as_external'];
+            }
+          }
           // Return array $data['sender_acl_domains/addresses']['ro'] with read-only objects
           // Return array $data['sender_acl_domains/addresses']['rw'] with read-write objects (can be deleted)
-          $stmt = $pdo->prepare("SELECT REPLACE(`send_as`, '@', '') AS `send_as` FROM `sender_acl` WHERE `logged_in_as` = :logged_in_as AND (`send_as` LIKE '@%' OR `send_as` = '*')");
+          $stmt = $pdo->prepare("SELECT REPLACE(`send_as`, '@', '') AS `send_as` FROM `sender_acl` WHERE `logged_in_as` = :logged_in_as AND `external` = '0' AND (`send_as` LIKE '@%' OR `send_as` = '*')");
           $stmt->execute(array(':logged_in_as' => $_data));
           $domain_rows = $stmt->fetchAll(PDO::FETCH_ASSOC);
           while ($domain_row = array_shift($domain_rows)) {
@@ -2344,15 +2496,15 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               $data['sender_acl_domains']['rw'][] = $domain_row['send_as'];
             }
           }
-          $stmt = $pdo->prepare("SELECT `send_as` FROM `sender_acl` WHERE `logged_in_as` = :logged_in_as AND (`send_as` NOT LIKE '@%' AND `send_as` != '*')");
+          $stmt = $pdo->prepare("SELECT `send_as` FROM `sender_acl` WHERE `logged_in_as` = :logged_in_as AND `external` = '0' AND (`send_as` NOT LIKE '@%' AND `send_as` != '*')");
           $stmt->execute(array(':logged_in_as' => $_data));
           $address_rows = $stmt->fetchAll(PDO::FETCH_ASSOC);
           while ($address_row = array_shift($address_rows)) {
-            if (filter_var($address_row['send_as'], FILTER_VALIDATE_EMAIL) && !hasMailboxObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $address_row['send_as'])) {
+            if (filter_var($address_row['send_as'], FILTER_VALIDATE_EMAIL) && !hasAliasObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $address_row['send_as'])) {
               $data['sender_acl_addresses']['ro'][] = $address_row['send_as'];
               continue;
             }
-            if (filter_var($address_row['send_as'], FILTER_VALIDATE_EMAIL) && hasMailboxObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $address_row['send_as'])) {
+            if (filter_var($address_row['send_as'], FILTER_VALIDATE_EMAIL) && hasAliasObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $address_row['send_as'])) {
               $data['sender_acl_addresses']['rw'][] = $address_row['send_as'];
               continue;
             }
@@ -2361,12 +2513,14 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             WHERE `domain` NOT IN (
               SELECT REPLACE(`send_as`, '@', '') FROM `sender_acl` 
                 WHERE `logged_in_as` = :logged_in_as1
+                  AND `external` = '0'
                   AND `send_as` LIKE '@%')
             UNION
             SELECT '*' FROM `domain`
               WHERE '*' NOT IN (
                 SELECT `send_as` FROM `sender_acl`  
                   WHERE `logged_in_as` = :logged_in_as2
+                    AND `external` = '0'
               )");
           $stmt->execute(array(
             ':logged_in_as1' => $_data,
@@ -2388,6 +2542,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               AND `address` NOT IN (
                 SELECT `send_as` FROM `sender_acl` 
                   WHERE `logged_in_as` = :logged_in_as
+                    AND `external` = '0'
                     AND `send_as` NOT LIKE '@%')");
           $stmt->execute(array(
             ':logged_in_as' => $_data,
@@ -2395,7 +2550,11 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           ));
           $rows_mbox = $stmt->fetchAll(PDO::FETCH_ASSOC);
           while ($row = array_shift($rows_mbox)) {
-            if (filter_var($row['address'], FILTER_VALIDATE_EMAIL) && hasMailboxObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $row['address'])) {
+            // Aliases are not selectable
+            if (in_array($row['address'], $data['fixed_sender_aliases'])) {
+              continue;
+            }
+            if (filter_var($row['address'], FILTER_VALIDATE_EMAIL) && hasAliasObjectAccess($_SESSION['mailcow_cc_username'], $_SESSION['mailcow_cc_role'], $row['address'])) {
               $data['sender_acl_addresses']['selectable'][] = $row['address'];
             }
           }
@@ -2852,7 +3011,13 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             ':aliasdomain' => $_data,
           ));
           $row = $stmt->fetch(PDO::FETCH_ASSOC);
+          $stmt = $pdo->prepare("SELECT `backupmx` FROM `domain` WHERE `domain` = :target_domain");
+          $stmt->execute(array(
+            ':target_domain' => $row['target_domain']
+          ));
+          $row_parent = $stmt->fetch(PDO::FETCH_ASSOC);
           $aliasdomaindata['alias_domain'] = $row['alias_domain'];
+          $aliasdomaindata['parent_is_backupmx'] = $row_parent['backupmx'];
           $aliasdomaindata['target_domain'] = $row['target_domain'];
           $aliasdomaindata['active'] = $row['active'];
           $aliasdomaindata['rl'] = $rl;
@@ -2904,6 +3069,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               `description`,
               `aliases`,
               `mailboxes`, 
+              `defquota`,
               `maxquota`,
               `quota`,
               `relayhost`,
@@ -2935,6 +3101,10 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           if ($domaindata['max_new_mailbox_quota'] > ($row['maxquota'] * 1048576)) {
             $domaindata['max_new_mailbox_quota'] = ($row['maxquota'] * 1048576);
           }
+          $domaindata['def_new_mailbox_quota'] = $domaindata['max_new_mailbox_quota'];
+          if ($domaindata['def_new_mailbox_quota'] > ($row['defquota'] * 1048576)) {
+            $domaindata['def_new_mailbox_quota'] = ($row['defquota'] * 1048576);
+          }
           $domaindata['quota_used_in_domain'] = $MailboxDataDomain['in_use'];
           $domaindata['mboxes_in_domain'] = $MailboxDataDomain['count'];
           $domaindata['mboxes_left'] = $row['mailboxes']	- $MailboxDataDomain['count'];
@@ -2942,6 +3112,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           $domaindata['description'] = $row['description'];
           $domaindata['max_num_aliases_for_domain'] = $row['aliases'];
           $domaindata['max_num_mboxes_for_domain'] = $row['mailboxes'];
+          $domaindata['def_quota_for_mbox'] = $row['defquota'] * 1048576;
           $domaindata['max_quota_for_mbox'] = $row['maxquota'] * 1048576;
           $domaindata['max_quota_for_domain'] = $row['quota'] * 1048576;
           $domaindata['relayhost'] = $row['relayhost'];
@@ -3006,7 +3177,14 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             $mailboxdata['max_new_quota'] = ($DomainQuota['maxquota'] * 1048576);
           }
           $mailboxdata['username'] = $row['username'];
-          $mailboxdata['rl'] = $rl;
+          if (!empty($rl)) {
+            $mailboxdata['rl'] = $rl;
+            $mailboxdata['rl_scope'] = 'mailbox';
+          }
+          else {
+            $mailboxdata['rl'] = ratelimit('get', 'domain', $row['domain']);
+            $mailboxdata['rl_scope'] = 'domain';
+          }
           $mailboxdata['is_relayed'] = $row['backupmx'];
           $mailboxdata['name'] = $row['name'];
           $mailboxdata['active'] = $row['active'];
@@ -3016,10 +3194,13 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
           $mailboxdata['quota'] = $row['quota'];
           $mailboxdata['attributes'] = json_decode($row['attributes'], true);
           $mailboxdata['quota_used'] = intval($row['bytes']);
-          $mailboxdata['percent_in_use'] = round((intval($row['bytes']) / intval($row['quota'])) * 100);
+          $mailboxdata['percent_in_use'] = ($row['quota'] == 0) ? '- ' : round((intval($row['bytes']) / intval($row['quota'])) * 100);
           $mailboxdata['messages'] = $row['messages'];
           $mailboxdata['spam_aliases'] = $SpamaliasUsage['sa_count'];
-          if ($mailboxdata['percent_in_use'] >= 90) {
+          if ($mailboxdata['percent_in_use'] === '- ') {
+            $mailboxdata['percent_class'] = "info";
+          }
+          elseif ($mailboxdata['percent_in_use'] >= 90) {
             $mailboxdata['percent_class'] = "danger";
           }
           elseif ($mailboxdata['percent_in_use'] >= 75) {
@@ -3317,7 +3498,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
               $_SESSION['return'][] = array(
                 'type' => 'danger',
                 'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
-                'msg' => 'domain_not_empty'
+                'msg' => array('domain_not_empty', $domain)
               );
               continue;
             }
@@ -3411,6 +3592,10 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             $stmt->execute(array(
               ':id' => $id
             ));
+            $stmt = $pdo->prepare("DELETE FROM `sender_acl` WHERE `send_as` = :alias_address");
+            $stmt->execute(array(
+              ':alias_address' => $alias_data['address']
+            ));
             $_SESSION['return'][] = array(
               'type' => 'success',
               'log' => array(__FUNCTION__, $_action, $_type, $_data_log, $_attr),
@@ -3525,7 +3710,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             }
             if (strtolower(getenv('SKIP_SOLR')) == 'n') {
               $curl = curl_init();
-              curl_setopt($curl, CURLOPT_URL, 'http://solr:8983/solr/dovecot/update?commit=true');
+              curl_setopt($curl, CURLOPT_URL, 'http://solr:8983/solr/dovecot-fts/update?commit=true');
               curl_setopt($curl, CURLOPT_HTTPHEADER,array('Content-Type: text/xml'));
               curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
               curl_setopt($curl, CURLOPT_POST, 1);
@@ -3587,7 +3772,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
             $stmt->execute(array(
               ':username' => $username
             ));
-            $stmt = $pdo->prepare("DELETE FROM `sogo_acl` WHERE `c_object` LIKE '%/" . $username . "/%' OR `c_uid` = :username");
+            $stmt = $pdo->prepare("DELETE FROM `sogo_acl` WHERE `c_object` LIKE '%/" . str_replace('%', '\%', $username) . "/%' OR `c_uid` = :username");
             $stmt->execute(array(
               ':username' => $username
             ));
@@ -3714,7 +3899,7 @@ function mailbox($_action, $_type, $_data = null, $_extra = null) {
       }
     break;
   }
-  if ($_action != 'get' && in_array($_type, array('domain', 'alias', 'alias_domain', 'mailbox'))) {
+  if ($_action != 'get' && in_array($_type, array('domain', 'alias', 'alias_domain', 'mailbox', 'resource'))) {
     update_sogo_static_view();
   }
 }