Provider Migrations

This document describes the steps for migrating openDesk from one cloud service provider to another. For the sake of simplicity this document assumes that the Kubernetes-Cluster is an off-the-shelf cluster of said CSP, hence no K8S-Migration occurs. We also assume that in-cluster persistence services are used (e.g. S3, PostgreSQL,…). CSP-Services may behave differently but follow similar patterns.

Guidelines & Suggestions

The following values have proven effective in practice. Adjust based on your infrastructure and requirements.

Timeline & Overview

What to Migrate

1. Configuration Files

2. Database Credentials

Database credentials in openDesk work as follows:

1. Technical usernames (e.g., keycloak_user) are defined in databases.yaml.gotmpl
2. Database passwords are derived from MASTER_PASSWORD via the Master Password algorithm in secrets.yaml.gotmpl
3. To migrate: Use the same MASTER_PASSWORD in target deployment, then migrate only the actual data

3. Managed Databases (CSP Services)

openDesk uses managed database services (PostgreSQL, MariaDB) from the CSP:

Migration approach:

• PostgreSQL: Use pg_dump via port-forward for logical exports
• MariaDB: Use mysqldump via port-forward for logical exports
• Cassandra (EE-only): Export using nodetool snapshot or cqlsh COPY commands
• Redis: Not typically backed up (cache only); see Data Storage documentation

Note: For CSP-managed databases, ensure the target service is created with compatible versions before data migration.

4. Physical Volumes (PVCs)

Application PersistentVolumeClaims store user data and application state:

Migration approach: Use kubectl get pvc -o yaml to export PVC definitions, then use storage provider tools or kubectl cp to migrate data.

Alternative: For storage provider migrations, consider using storage-specific migration tools or CSI driver features.

5. LDAP Data

Export OpenLDAP directory from the ums-ldap-server-primary service.

Secret Service: ums-ldap-server-admin (password key)

Base DN: Get from pod environment (LDAP_BASE_DN), default: dc=swp-ldap,dc=internal

6. Object Storage (S3 Buckets)

Migration approach: Use s3 sync with server-side encryption considerations.

7. Redis Cache

Redis instances are used for session and data caching (Nextcloud, OX App Suite, OpenProject). Redis data is ephemeral and typically not backed up.

If you need to preserve session state during migration:

kubectl exec -n ${NAMESPACE} redis-master-0 -- redis-cli SAVE
kubectl exec -n ${NAMESPACE} redis-master-0 -- cat /data/dump.rdb

8. Kubernetes Secrets

Migrate all Kubernetes Secrets from the source namespace to ensure credentials, API keys, and certificates are available in the target cluster.

Alternatively, use the K8up approach:

• The PreBackupPod secrets-pre-backup-pod exports all secrets as JSON
• This can be restored using K8up Restore CR

9. DNS Records

For detailed backup strategies including K8up, Restic, and rclone configurations, see the Backup & Recovery documentation.

Migration Steps

Step 1: Reduce DNS TTLs (days before)

Consider setting DNS TTL to ~300 seconds for faster propagation and easier rollback.

Many openDesks are managed by delegated DNS-Zones. Zone-TTLs are often quite high. Depending on what exactly is migrated ( zone or record ) the shortened TTL may be the zone or the records.

Step 2: Build New Infrastructure

1. Use existing configuration files to deploy openDesk on new provider. openDesk is domain-centric, hence a Test-Deployment on a different domain may be used to verify the deployment generally works but needs to be re-deployed with the correct domain for a later migration.
2. Verify base deployment works.

Most openDesk setups require LetsEncrypt for certificate handling. Transfer the LE Secrets. Alternatively the DNS-Challenge may issue valid certificates to an offline-cluster.

Step 3: Maintenance Window - Disable Ingress

Consider disabling all ingresses to prevent new write traffic during migration.

Step 4: Export All Data

Export all databases, LDAP, object storage, PVCs, and Kubernetes Secrets to the migration S3 bucket.

Key export commands:

• PostgreSQL: pg_dump via port-forward
• MariaDB: mysqldump via port-forward
• Cassandra: nodetool snapshot or cqlsh COPY
• LDAP: ldapsearch via port-forward
• Object Storage: aws s3 sync
• PVCs: kubectl get pvc -o yaml + storage provider tools
• Secrets: kubectl get secrets -o json

Step 5: Import All Data

Restore all databases, LDAP, object storage, PVCs, and Kubernetes Secrets to the target cluster.

Key import commands:

• PostgreSQL: pg_restore via port-forward
• MariaDB: mysql via port-forward
• Cassandra: nodetool refresh or cqlsh COPY
• LDAP: slapadd (stop StatefulSet first)
• Object Storage: aws s3 sync
• PVCs: Recreate PVCs and restore data
• Secrets: kubectl apply -f

Step 6: Nameserver Cutover

The nameserver switch approach allows you to prepare DNS changes before the migration window, significantly reducing downtime.

DNS Records to Migrate

Copy SPF Record

1. From: Query current DNS at old provider
   dig TXT yourdomain.com +short
   # or
   nslookup -type=TXT yourdomain.com

2. To: Add identical TXT record at new DNS provider
3. Verify: dig TXT yourdomain.com +short after propagation

Copy DKIM Record

1. From: Extract public key from OpenDKIM pod or Keycloak
   # Option A: From OpenDKIM pod
   kubectl exec -n ${SOURCE_NS} opendkim-0 -c main -- cat /etc/opendkim/keys/${DOMAIN}/mail.txt
    
   # Option B: From Keycloak realm settings (if DKIM is managed there)
   # Check realm settings in Keycloak admin UI under Email -> DKIM

2. To: Add as TXT record at new DNS provider
   • Selector: match what your current _domainkey.yourdomain.com shows (often default or mail)
   • Content: The full v=DKIM1; p=... string
3. Verify: dig TXT default._domainkey.yourdomain.com +short

Copy DMARC Record

1. From: Query current DNS
   dig TXT _dmarc.yourdomain.com +short

2. To: Add identical TXT record at new DNS provider with prefix _dmarc
3. Verify: dig TXT _dmarc.yourdomain.com +short

Execution Order

1. Before migration window (preparation): Copy all records to new DNS provider but keep old nameservers active
2. During migration window: Switch authoritative nameservers to new provider
3. Wait for TTL propagation (typically 5-30 minutes)

Step 7: Post-Migration Restart

Consider restarting applications in a logical order (databases first, then services) to ensure proper initialization.

Step 8: Verification

Consider testing using DNS-resolved hostname or test accounts.

Test Accounts

Local DNS Testing

For testing before DNS propagation, add entries to /etc/hosts:

<CLUSTER_IP> files.yourdomain.de
<CLUSTER_IP> ics.yourdomain.de
<CLUSTER_IP> id.yourdomain.de
<CLUSTER_IP> objectstore.yourdomain.de
<CLUSTER_IP> dav.yourdomain.de
<CLUSTER_IP> webmail.yourdomain.de
<CLUSTER_IP> yourdomain.de
<CLUSTER_IP> adminbot.yourdomain.de
<CLUSTER_IP> office.yourdomain.de
<CLUSTER_IP> projects.yourdomain.de
<CLUSTER_IP> chat.yourdomain.de
<CLUSTER_IP> meet.yourdomain.de
<CLUSTER_IP> matrix.yourdomain.de
<CLUSTER_IP> wiki.yourdomain.de
<CLUSTER_IP> pad.yourdomain.de
<CLUSTER_IP> portal.yourdomain.de

Step 9: Open to Users

1. Allow access to new environment
2. Monitor for issues
3. Consider keeping source cluster running for 7-14 days (rollback option)

Verification Checklist

Authentication & Identity

Mail System

Applications

Federation & Integrations

Data Integrity & DNS

Migration-Specific Tests

Matrix Neodate Bot Recovery

If the matrix-neodatefix-bot pod fails to start after migration with M_UNKNOWN_TOKEN error, the Matrix credentials need to be regenerated:

# Delete the affected secrets and helm charts
cd helmfile/apps/element
helmfile destroy -l name=matrix-neodatefix-bot-bootstrap
helmfile destroy -l name=opendesk-matrix-user-verification-bootstrap
 
# Redeploy to regenerate credentials
helmfile apply -l name=matrix-neodatefix-bot-bootstrap
helmfile apply -l name=opendesk-matrix-user-verification-bootstrap

RTO Calculation

Cloud Interconnect (1 Gbps) migration times with safety factor (1.4×) and overhead (~45 min):

Formula used: ((Data_Size_GB × 8) / 1000) × 2 × 1.4 + 0.75 (hours)

Rollback

If migration fails:

1. Keep DNS pointing to source
2. Keep source cluster running
3. Investigate and retry
4. Use K8up Restore jobs to re-sync any partial data transfers