ungleich blog

IPv6 only email services coming soon

2024-10-19T00:00:00Z

TL;DR

Soon we will offer IPv6 only email services. So far it has been decided that the SMTP part will be IPv6 only, the IMAP/webmail part might be dual stack or IPv6 only as well.

Next generation services - going one step further

When you know us, you know that we have been pushing IPv6 only services since 2017. IPv6 only VMs - we got it. IPv6 only websites - we got it. IPv6 only routers and networks? We got plenty of them.

Now we played a bit with the though of, what if there was an email domain that can only receive emails from IPv6 capable hosts? It would certainly be a challenge, but more for the legacy only providers, not for us.

So stay tuned and expect an IPv6 only email service soon to be available.

Our staticcms has been moved to kubernetes!

2024-10-13T00:00:00Z

Moved to kubernetes

This website and all static content has been moved to Kubernetes as of 2024-10-13. The motivation and background is very similar to what Nico wrote on his blog already some days ago.

Build system for teams

Differently to what Nico is doing with his private homepage, this website will be moved into a "fancy build" system. The background is that while for a single person running make on the laptop which modifies argocd cluster configurations, for a team of people working on the website, things are a bit more difficult.

For once, not every computer at ungleich is running Linux, we do have some non-Linux machines and also machines not run by our kubernetes team, so there is no direct argocd access. And neither is docker build available on some machines.

How the build pipeline will look exactly is still to be defined, but once we have settled on one, you will link the details here - so stay tuned for updates.

Static websites pattern

One thing we would like to share is already the structure of the helm chart we are using for deployment: We have one helm chart that uses a list of containers that should be deployed. Deployed by argocd it essentially looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staticweb-nico
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: nico
    server: 'https://kubernetes.default.svc'
  source:
    path: apps/prod/staticweb
    repoURL: '...'
    helm:
      valuesObject:
        websites:
          - name: www-nico
            image:
              harbor.k8s.ungleich.ch/nico/www.nico.schottelius.org:559c0c45
            fqdn: www.nico.schottelius.org
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The name attribute is used for defining the k8s Deployment, the fqdn is passed to the nginx-ingress and the image should be pretty clear.

You may notice that we use our own image registry based on harbor. Not only is this much faster for pulling images, but it also solves problems with other registries:

github (ghrc.io) is legacy IP only and cannot be used from the Internet
docker hub has very strict pull limits, often not allowing us to pull in images. While this can be worked around with using a login token, it add a certain complexity to our infrastructure deploying logins for various registries.

This brings us to something some of you might have noticed already:

Our containers are publicly reachable

Anyone can download any container hosted on our harbor instance without authentication. This is by design, as containers should only contain public data. All our containers contain public, open source applications only.

Data is being injected either by an sqldump in the beginning (if it is an existing application) or generated at startup or passed in as a secret independently of the container.

From our perspective this principle allows easy and strict separation of private vs. public data.

cdist does not (yet) fully support OpenWrt

2024-08-02T00:00:00Z

cdist config management

As many of you know, we use cdist for configuration management at ungleich. And we try to manage everything that is not (yet) in kubernetes with it.

Today's short blog entry is about using cdist with openwrt.

OpenWrt at ungleich

[OpenWrt](https://openwrt.org] is a popular, open source operating system used on routers, switches, etc. We use it for providing IPv6 and IPv4 connectivity to customers world wide.

So far most of our devices are configured using shell scripts from our ungleich-tool git repository.

However as configurations get more complex, we thought about moving our configuration also into cdist.

OpenWrt is similar, but not the same

OpenWrt is, generally speaking, "just another Linux distribution", albeit with a very, very small footprint. It has to be, because the storage on a typical router can be in the size of a couple Megabytes. Right, not Gigabyte, not Terabyte.

For instance this "large" router has about 9 Megabytes of storage:

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.0M      4.0M         0 100% /rom
tmpfs                   217.0M    240.0K    216.8M   0% /tmp
/dev/mtdblock6            9.1M    420.0K      8.7M   4% /overlay
overlayfs:/overlay        9.1M    420.0K      8.7M   4% /
tmpfs                   512.0K         0    512.0K   0% /dev

It does however have 512 MiB of RAM...:

# free -m
              total        used        free      shared  buff/cache   available
Mem:         444428       57120      370184         240       17124      354048
Swap:             0           0           0

So, openwrt is a Linux distribution, but its environment is a bit more challenging than a general purpose Linux distribution.

cdist & openwrt

Cdist uses so called "types" to configure systems idempotently. They usually require some shell support on the target system, but nothing fancy.

However in the case of openwrt, cdist is missing some support, as of version 7.0.0:

cdist uses the system default ssh and if that uses SFTP instead of SCP by default, it will fail, as openwrt, as of 23.05.3, only supports legacy scp (-O). Manually patching cdist source code to include "-O" fixes this issue for the moment.
the most basic cdist type __file uses "cksum" to create a checksum over files to decide whether or not to copy a file. cksum was chosen in the first place, as it is very basic and can be found everywhere. Well, everywhere but on openwrt...

Future of cdist & openwrt

Both above issues can in theory be addressed, but the __file type is very basic and needs some checksumming support. Without it, cdist lacks a major feature and can be considered not (yet) usable.

Let's see what the future brings.

Migrating from Twitter to Mastodon

2022-11-11T00:00:00Z

TL;DR

If you are in a hurry, here is the minimal amount of information you need to move from Twitter to Mastodon:

Find a server you like
Create an account
Follow people you like

Ready and go!

The longer story

Mastodon is not Twitter and will never be. However both Mastodon and Twitter are social media networks. Mastodon however is decentralised.

Decentralised Social Media

What does that mean, decentralised? It means anyone can run a Mastodon server and connect to everyone else. This is very much how the Internet was designed: a place of many systems that talk to each other.

Twitter on the other side is a centralised solution. All data and communication is hosted by Twitter. If Twitter is going down, everyone is affected.

Mastodon on the other hand is much more robust. You can and have to choose a Mastodon instance that you like.

Which Mastodon instance to use?

There are plenty of instances around and you can choose the one you trust most or your friends are on. There are various lists of Mastodon instances, such as the one at joinmastodon.org or at instances.social (thanks to @AltoidLover@tech.lgbt for the pointer).

Or, you can even run your own Mastodon instance.

Your own Mastodon instance

Why would you want to run your own Mastodon instance in the first place? Let's take a few steps back, to understand the fundamental difference between Twitter and the Fediverse. The Fedi-what?

We so far said Mastodon everywhere, but in reality, Mastodon is just one software that allows you to communicate decentralised. In general, we refer to the decentralised communication as the "FEDIVERSE". That's settled, now let's go back to why anyone would want to run their own instance.

First of all, running your own mastodon instance allows you to use your own, custom domain. So my handle is @nico@ipv6.social and it indicates I am somewhat interested in #IPv6 related topics, just from the domain of the instance I am on.

If you already have a domain, many people also choose to use the subdomain "social", like social.example.com.

There is a second, maybe equally significant point when it comes to Mastodon. That is the question of who is going to take care of the instance, who is going to update the software and who is actually going to pay for running the instance?

In the fediverse, you are not a product

Compared to Twitter, where advertisement turns the user into a product, in the fediverse you are not a product. That also means that the organisation running a Mastodon instance does actually need to pay for the service. Whether that is running a VPS, running a Raspberry PI or having it hosted by someone else, somebody has to provide the resource for it.

This is fundamentally different compared to Twitter, but also Facebook, where you are basically also just a product.

The big advantage of this is that there is no advertisement, no central user analytics, but much more freedom. Like we say in Switzerland, you Khasch nit z'Füferli und z Weggli ha or in proper English: you cannot have everything. And that's a good thing.

Importing Twitter Followers into the Fediverse

You might have noticed that more and more Twitter accounts add their fediverse handle like @ungleich@ipv6.social to their Twitter handle. There is a nifty tool available that scans your account for the handles and imports them on the fediverse site for you. It is called Debirdify and it's a good start to get a list of people to follow. Debirdify can actually scan your followers and the ones you follow, too.

More guides

If you interested in reading Is there more about moving from Twitter to Mastodon, there are other articles you might be interested in, like:

A beginner’s guide to Mastodon, the open source Twitter alternative (techcrunch)
How-To Mastodon (Tobru, covers Swiss specific aspects) (thanks for @tobru@mstdn.social for the pointer)

Mastodon Hosting by ungleich

Shameless plug at the end - we have just launched Mastodon hosting by ungleich to provide you an easy way to start in the fediverse. Remember, you don't have to go with us, but the beauty of the fediverse is you can host your own, use somebody else's server or use a hosted Mastodon instance.

P.S.: A Post Twitter World

Twitter might not yet be done and even at ungleich we have a lot of references to our Twitter account. We don't know whether everyone will move to the fediverse, but we very much welcome a push towards decentralisation of the Internet.

For many years, probably about a decade or two, we have become more and more reliant on centralised services. These services might have made our lives easier, but also much more dependent on a few organisations.

It is a good thing to review what we are using from time to time and today we are looking stronger at Twitter - tomorrow we might consider a different search engine - who knows?

I am looking forward to hearing you in the fediverse and if you have any comment, do not hesitate to reach out to @nico@ipv6.social.

[WIP] Migrating Ceph Nautilus into Kubernetes + Rook

2022-08-27T00:00:00Z

Introduction

At ungleich we are running multiple Ceph clusters. Some of them are running Ceph Nautilus (14.x) based on Devuan. Our newer Ceph Pacific (16.x) clusters are running based on Rook on Kubernetes on top of Alpine Linux.

In this blog article we will describe how to migrate Ceph/Native/Devuan to Ceph/k8s+rook/Alpine Linux.

Work in Progress [WIP]

This blog article is work in progress. The migration planning has started, however the migration has not been finished yet. This article will feature the different paths we take for the migration.

The Plan

To continue operating the cluster during the migration, the following steps are planned:

Setup a k8s cluster that can potentially communicate with the existing ceph cluster
Using the disaster recovery guidelines from rook to modify the rook configuration to use the previous fsid.
Spin up ceph monitors and ceph managers in rook
Retire existing monitors
Shutdown a ceph OSD node, remove it's OS disk, boot it with Alpine Linux
Join the node into the k8s cluster
Have rook pickup the existing disks and start the osds
Repeat if successful
Migrate to ceph pacific

Original cluster

The target ceph cluster we want to migrate lives in the 2a0a:e5c0::/64 network. Ceph is using:

public network  = 2a0a:e5c0:0:0::/64
cluster network = 2a0a:e5c0:0:0::/64

Kubernetes cluster networking inside the ceph network

To be able to communicate with the existing OSDs, we will be using sub networks of 2a0a:e5c0::/64 for kubernetes. As these networks are part of the link assigned network 2a0a:e5c0::/64, we will use BGP routing on the existing ceph nodes to create more specific routes into the kubernetes cluster.

As we plan to use either cilium or calico as the CNI, we can configure kubernetes to directly BGP peer with the existing Ceph nodes.

The setup

Kubernetes Bootstrap

As usual we bootstrap 3 control plane nodes using kubeadm. The proxy for the API resides in a different kuberentes cluster.

We run

kubeadm init --config kubeadm.yaml

on the first node and join the other two control plane nodes. As usual, joining the workers last.

k8s Networking / CNI

For this setup we are using calico as described in the ungleich kubernetes manual.

VERSION=v3.23.3
helm repo add projectcalico https://docs.projectcalico.org/charts
helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace

BGP Networking on the old nodes

To be able to import the BGP routes from Kubernetes, all old / native hosts will run bird. The installation and configuration is as follows:

apt-get update
apt-get install -y bird2

router_id=$(hostname | sed 's/server//')

cat > /etc/bird/bird.conf <<EOF

router id $router_id;

log syslog all;
protocol device {
}
 # We are only interested in IPv6, skip another section for IPv4
protocol kernel {
        ipv6 { export all; };
}
protocol bgp k8s {
        local     as 65530;
        neighbor range 2a0a:e5c0::/64 as 65533;
        dynamic name "k8s_"; direct;

        ipv6 {
            import filter { if net.len > 64 then accept; else reject; };
            export none;
        };
}
EOF
/etc/init.d/bird restart

The router id must be adjusted for every host. As all hosts have a unique number, we use that one as the router id. The bird configuration allows to use dynamic peers so that any k8s node in the network can peer with the old servers.

We also use a filter to avoid receiving /64 routes, as they are overlapping with the on link route.

BGP networking in Kubernetes

Calico supports BGP peering and we use a rather standard calico configuration:

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: true
  asNumber: 65533
  serviceClusterIPs:
  - cidr: 2a0a:e5c0:0:aaaa::/108
  serviceExternalIPs:
  - cidr: 2a0a:e5c0:0:aaaa::/108

Plus for each server and router we create a BGPPeer:

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: serverXX
spec:
  peerIP: 2a0a:e5c0::XX
  asNumber: 65530
  keepOriginalNextHop: true

We apply the whole configuration using calicoctl:

./calicoctl create -f - < ~/vcs/k8s-config/bootstrap/p5-cow/calico-bgp.yaml

And a few seconds later we can observer the routes on the old / native hosts:

bird> show protocols
Name       Proto      Table      State  Since         Info
device1    Device     ---        up     23:09:01.393
kernel1    Kernel     master6    up     23:09:01.393
k8s        BGP        ---        start  23:09:01.393  Passive
k8s_1      BGP        ---        up     23:33:01.215  Established
k8s_2      BGP        ---        up     23:33:01.215  Established
k8s_3      BGP        ---        up     23:33:01.420  Established
k8s_4      BGP        ---        up     23:33:01.215  Established
k8s_5      BGP        ---        up     23:33:01.215  Established

Testing networking

To verify that the new cluster is working properly, we can deploy a tiny test deployment and see if it is globally reachable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.20.0-alpine
        ports:
        - containerPort: 80

And the corresponding service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80

Using curl to access a sample service from the outside shows that networking is working:

% curl -v http://[2a0a:e5c0:0:aaaa::e3c9]
*   Trying 2a0a:e5c0:0:aaaa::e3c9:80...
* Connected to 2a0a:e5c0:0:aaaa::e3c9 (2a0a:e5c0:0:aaaa::e3c9) port 80 (#0)
> GET / HTTP/1.1
> Host: [2a0a:e5c0:0:aaaa::e3c9]
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.20.0
< Date: Sat, 27 Aug 2022 22:35:49 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 20 Apr 2021 16:11:05 GMT
< Connection: keep-alive
< ETag: "607efd19-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host 2a0a:e5c0:0:aaaa::e3c9 left intact

So far we have found 1 issue:

Sometimes the old/native servers can reach the service, sometimes they get a timeout

In old calico notes on github it is referenced that overlapping Pod/CIDR networks might be a problem. Additionally we cannot use kubeadm to initialise the podsubnet to be a proper subnet of the node subnet:

[00:15] server57.place5:~# kubeadm init --service-cidr 2a0a:e5c0:0:cccc::/108 --pod-network-cidr 2a0a:e5c0::/100
I0829 00:16:38.659341   19400 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.24
podSubnet: Invalid value: "2a0a:e5c0::/100": the size of pod subnet with mask 100 is smaller than the size of node subnet with mask 64
To see the stack trace of this error execute with --v=5 or higher
[00:16] server57.place5:~#

Networking 2022-09-03

Instead of trying to merge the cluster networks, we will use separate ranges
According to the ceph users mailing list discussion it is actually not necessary for mons/osds to be in the same network. In fact, we might be able to remove these settings completely.

So today we start with

podSubnet: 2a0a:e5c0:0:14::/64
serviceSubnet: 2a0a:e5c0:0:15::/108

Using BGP and calico, the kubernetes cluster is setup "as usual" (for ungleich terms).

Ceph.conf change

Originally our ceph.conf contained:

public network  = 2a0a:e5c0:0:0::/64
cluster network = 2a0a:e5c0:0:0::/64

As of today they are removed and all daemons are restarted, allowing the native cluster to speak with the kubernetes cluster.

Setting up rook

Usually we deploy rook via argocd. However as we want to be easily able to do manual intervention, we will first bootstrap rook via helm directly and turn off various services

helm repo add rook https://charts.rook.io/release
helm repo update

We will use rook 1.8, as it is the last version to support Ceph nautilus, which is our current ceph version. The latest 1.8 version is 1.8.10 at the moment.

helm upgrade --install --namespace rook-ceph --create-namespace --version v1.8.10 rook-ceph rook/rook-ceph

Joining the 2 clusters, step 1: monitors and managers

In the first step we want to add rook based monitors and managers and replace the native ones. For rook to be able to talk to our existing cluster, it needs to know

the current monitors/managers ("the monmap")
the right keys to talk to the existing cluster
the fsid

As we are using v1.8, we will follow the guidelines for disaster recover of rook 1.8.

Later we will need to create all the configurations so that rook knows about the different pools.

Rook: CephCluster

Rook has a configuration of type CephCluster that typically looks something like this:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    # see the "Cluster Settings" section below for more details on which image of ceph to run
    image: quay.io/ceph/ceph:{{ .Chart.AppVersion }}
  dataDirHostPath: /var/lib/rook
  mon:
    count: 5
    allowMultiplePerNode: false
  storage:
    useAllNodes: true
    useAllDevices: true
    onlyApplyOSDPlacement: false
  mgr:
    count: 1
    modules:
      - name: pg_autoscaler
        enabled: true
  network:
    ipFamily: "IPv6"
    dualStack: false
  crashCollector:
    disable: false
    # Uncomment daysToRetain to prune ceph crash entries older than the
    # specified number of days.
    daysToRetain: 30

For migrating, we don't want rook in the first stage to create any OSDs. So we will replace useAllNodes: true with useAllNodes: false and useAllDevices: true also with useAllDevices: false.

Extracting a monmap

To get access to the existing monmap, we can export it from the native cluster using ceph-mon -i {mon-id} --extract-monmap {map-path}. More details can be found on the documentation for adding and removing ceph monitors.

Rook and Ceph pools

Rook uses CephBlockPool to describe ceph pools as follows:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: hdd
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
  deviceClass: hdd

In this particular cluster we have 2 pools:

one (ssd based, device class = ssd)
hdd (hdd based, device class = hdd-big)

The device class "hdd-big" is specific to this cluster as it used to contain 2.5" and 3.5" HDDs in different pools.

[old] Analysing the ceph cluster configuration

Taking the view from the old cluster, the following items are important for adding new services/nodes:

We have a specific fsid that needs to be known
- The expectation would be to find that fsid in a configmap/secret in rook
We have a list of running monitors
- This is part of the monmap and ceph.conf
- ceph.conf is used for finding the initial contact point
- Afterwards the information is provided by the monitors
- For rook it would be expected to have a configmap/secret listing the current monitors
The native clusters have a "ceph.client.admin.keyring" deployed which allows adding and removing resources.
- Rook probably has a secret for keyrings
- Maybe multiple depending on how services are organised

Analysing the rook configurations

Taking the opposite view, we can also checkout a running rook cluster and the rook disaster recovery documentation to identify what to modify.

Let's have a look at the secrets first:

cluster-peer-token-rook-ceph                 kubernetes.io/rook                    2      320d
default-token-xm9xs                          kubernetes.io/service-account-token   3      320d
rook-ceph-admin-keyring                      kubernetes.io/rook                    1      320d
rook-ceph-admission-controller               kubernetes.io/tls                     3      29d
rook-ceph-cmd-reporter-token-5mh88           kubernetes.io/service-account-token   3      320d
rook-ceph-config                             kubernetes.io/rook                    2      320d
rook-ceph-crash-collector-keyring            kubernetes.io/rook                    1      320d
rook-ceph-mgr-a-keyring                      kubernetes.io/rook                    1      320d
rook-ceph-mgr-b-keyring                      kubernetes.io/rook                    1      320d
rook-ceph-mgr-token-ktt2m                    kubernetes.io/service-account-token   3      320d
rook-ceph-mon                                kubernetes.io/rook                    4      320d
rook-ceph-mons-keyring                       kubernetes.io/rook                    1      320d
rook-ceph-osd-token-8m6lb                    kubernetes.io/service-account-token   3      320d
rook-ceph-purge-osd-token-hznnk              kubernetes.io/service-account-token   3      320d
rook-ceph-rgw-token-wlzbc                    kubernetes.io/service-account-token   3      134d
rook-ceph-system-token-lxclf                 kubernetes.io/service-account-token   3      320d
rook-csi-cephfs-node                         kubernetes.io/rook                    2      320d
rook-csi-cephfs-plugin-sa-token-hkq2g        kubernetes.io/service-account-token   3      320d
rook-csi-cephfs-provisioner                  kubernetes.io/rook                    2      320d
rook-csi-cephfs-provisioner-sa-token-tb78d   kubernetes.io/service-account-token   3      320d
rook-csi-rbd-node                            kubernetes.io/rook                    2      320d
rook-csi-rbd-plugin-sa-token-dhhq6           kubernetes.io/service-account-token   3      320d
rook-csi-rbd-provisioner                     kubernetes.io/rook                    2      320d
rook-csi-rbd-provisioner-sa-token-lhr4l      kubernetes.io/service-account-token   3      320d

TBC

Creating additional resources after the cluster is bootstrapped

To let rook know what should be there, we already create the two CephBlockPool instances that match the existing pools:

```apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: one namespace: rook-ceph spec: failureDomain: host replicated: size: 3 deviceClass: ssd


And for the hdd based pool:

apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: hdd namespace: rook-ceph spec: failureDomain: host replicated: size: 3 deviceClass: hdd-big


Saving both of these in ceph-blockpools.yaml and applying it:

kubectl -n rook-ceph apply -f ceph-blockpools.yaml


### Configuring ceph after the operator deployment

As soon as the operator and the crds have been deployed, we deploy the
following
[CephCluster](https://rook.io/docs/rook/v1.8/ceph-cluster-crd.html)
configuration:

apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: quay.io/ceph/ceph:v14.2.21 dataDirHostPath: /var/lib/rook mon: count: 5 allowMultiplePerNode: false storage: useAllNodes: false useAllDevices: false onlyApplyOSDPlacement: false mgr: count: 1 modules:

  - name: pg_autoscaler
    enabled: true

network: ipFamily: "IPv6" dualStack: false crashCollector: disable: false

# Uncomment daysToRetain to prune ceph crash entries older than the
# specified number of days.
daysToRetain: 30


We wait for the cluster to initialise and stabilise before applying
changes. Important to note is that we use the ceph image version
v14.2.21, which is the same version as the native cluster.



### rook v1.8 is incompatible with ceph nautilus

After deploying the rook operator, the following error message is
printed in its logs:

2022-09-03 15:14:03.543925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed the ceph version check: the version does not meet the minimum version "15.2.0-0 octopus"


So we need to downgrade to rook v1.7. Using `helm search repo
rook/rook-ceph --versions` we identify the latest usable version should be `v1.7.11`.

We start the downgrade process using

helm upgrade --install --namespace rook-ceph --create-namespace --version v1.7.11 rook-ceph rook/rook-ceph


After downgrading the operator is starting the canary monitors and
continues to bootstrap the cluster.

### The ceph-toolbox

To be able to view the current cluster status, we also deploy the
ceph-toolbox for interacting with rook:

apiVersion: apps/v1 kind: Deployment metadata: name: rook-ceph-tools namespace: rook-ceph # namespace:cluster labels: app: rook-ceph-tools spec: replicas: 1 selector: matchLabels: app: rook-ceph-tools template: metadata: labels: app: rook-ceph-tools spec: dnsPolicy: ClusterFirstWithHostNet containers:

    - name: rook-ceph-tools
      image: rook/ceph:v1.7.11
      command: ["/bin/bash"]
      args: ["-m", "-c", "/usr/local/bin/toolbox.sh"]
      imagePullPolicy: IfNotPresent
      tty: true
      securityContext:
        runAsNonRoot: true
        runAsUser: 2016
        runAsGroup: 2016
      env:
        - name: ROOK_CEPH_USERNAME
          valueFrom:
            secretKeyRef:
              name: rook-ceph-mon
              key: ceph-username
        - name: ROOK_CEPH_SECRET
          valueFrom:
            secretKeyRef:
              name: rook-ceph-mon
              key: ceph-secret
      volumeMounts:
        - mountPath: /etc/ceph
          name: ceph-config
        - name: mon-endpoint-volume
          mountPath: /etc/rook
  volumes:
    - name: mon-endpoint-volume
      configMap:
        name: rook-ceph-mon-endpoints
        items:
          - key: data
            path: mon-endpoints
    - name: ceph-config
      emptyDir: {}
  tolerations:
    - key: "node.kubernetes.io/unreachable"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 5


### Checking the deployments

After the rook-operator finished deploying, the following deployments
are visible in kubernetes:

[17:25] blind:~% kubectl -n rook-ceph get deployment NAME READY UP-TO-DATE AVAILABLE AGE csi-cephfsplugin-provisioner 2/2 2 2 21m csi-rbdplugin-provisioner 2/2 2 2 21m rook-ceph-crashcollector-server48 1/1 1 1 2m3s rook-ceph-crashcollector-server52 1/1 1 1 2m24s rook-ceph-crashcollector-server53 1/1 1 1 2m2s rook-ceph-crashcollector-server56 1/1 1 1 2m17s rook-ceph-crashcollector-server57 1/1 1 1 2m1s rook-ceph-mgr-a 1/1 1 1 2m3s rook-ceph-mon-a 1/1 1 1 10m rook-ceph-mon-b 1/1 1 1 8m3s rook-ceph-mon-c 1/1 1 1 5m55s rook-ceph-mon-d 1/1 1 1 5m33s rook-ceph-mon-e 1/1 1 1 4m32s rook-ceph-operator 1/1 1 1 102m rook-ceph-tools 1/1 1 1 17m


Relevant for us are the mgr, mon and operator. To stop the cluster, we
will shutdown the deployments in the following order:

* rook-ceph-operator: to prevent deployments to recover

### Data / configuration comparison

Logging into a host that is running mon-a, we find the following data
in it:

[17:36] server56.place5:/var/lib/rook# find . ./mon-a ./mon-a/data ./mon-a/data/keyring ./mon-a/data/min_mon_release ./mon-a/data/store.db ./mon-a/data/store.db/LOCK ./mon-a/data/store.db/000006.log ./mon-a/data/store.db/000004.sst ./mon-a/data/store.db/CURRENT ./mon-a/data/store.db/MANIFEST-000005 ./mon-a/data/store.db/OPTIONS-000008 ./mon-a/data/store.db/OPTIONS-000005 ./mon-a/data/store.db/IDENTITY ./mon-a/data/kv_backend ./rook-ceph ./rook-ceph/crash ./rook-ceph/crash/posted ./rook-ceph/log


Which is pretty similar to the native nodes:

[17:37:50] red3.place5:/var/lib/ceph/mon/ceph-red3# find . ./sysvinit ./keyring ./min_mon_release ./kv_backend ./store.db ./store.db/1959645.sst ./store.db/1959800.sst ./store.db/OPTIONS-3617174 ./store.db/2056973.sst ./store.db/3617348.sst ./store.db/OPTIONS-3599785 ./store.db/MANIFEST-3617171 ./store.db/1959695.sst ./store.db/CURRENT ./store.db/LOCK ./store.db/2524598.sst ./store.db/IDENTITY ./store.db/1959580.sst ./store.db/2514570.sst ./store.db/1959831.sst ./store.db/3617346.log ./store.db/2511347.sst


### Checking how monitors are created on native ceph

To prepare for the migration we take 1 step back and verify how
monitors are created in the native cluster. The script used for
monitoring creation can be found on
[code.ungleich.ch](https://code.ungleich.ch/ungleich-public/ungleich-tools/src/branch/master/ceph/ceph-mon-create-start)
and contains the following logic:

* get "mon." key
* get the monmap
* Run ceph-mon --mkfs using the monmap and keyring
* Start it

In theory we could re-use these steps on a rook deployed monitor to
join our existing cluster.

### Checking the toolbox and monitor pods for migration

When the ceph-toolbox is deployed, we get a ceph.conf and a keyring in
/etc/ceph. The keyring is actually the admin keyring and allows us to
make modifications to the ceph cluster. The ceph.conf points to the
monitors and does not contain an fsid.

The ceph-toolbox gets this informatoin via 1 configmap
("rook-ceph-mon-endpoints") and a secret ("rook-ceph-mon").

The monitor pods on the other hand have an empty ceph.conf and no
admin keyring deployed.

### Try 1: recreating a monitor inside the existing cluster

Let's try to reuse an existing monitor and join it into the existing
cluster. For this we will first shut down the rook-operator, to
prevent it to intefere with our migration. Then
modify the relevant configmaps and secrets and import the settings
from the native cluster.

Lastly we will patch one of the monitor pods, inject the monmap from
the native cluster and then restart it.

Let's give it a try. First we shutdown the rook-ceph-operator:

% kubectl -n rook-ceph scale --replicas=0 deploy/rook-ceph-operator deployment.apps/rook-ceph-operator scaled


Then we patch the mon deployments to not run a monitor, but only
sleep:

for mon in a b c d e; do kubectl -n rook-ceph patch deployment rook-ceph-mon-${mon} -p \ '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}';

kubectl -n rook-ceph patch deployment rook-ceph-mon-$mon --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' done


Now the pod is restarted and when we execute into it, we will see that
no monitor is running in it:

% kubectl -n rook-ceph exec -ti rook-ceph-mon-a-c9f8f554b-2fkhm -- sh Defaulted container "mon" out of: mon, chown-container-data-dir (init), init-mon-fs (init) sh-4.2# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 4384 664 ? Ss 19:44 0:00 sleep infinity root 7 0.0 0.0 11844 2844 pts/0 Ss 19:44 0:00 sh root 13 0.0 0.0 51752 3384 pts/0 R+ 19:44 0:00 ps aux sh-4.2#


Now for this pod to work with our existing cluster, we want to import
the monmap and join the monitor to the native cluster. As with any
mon, the data is stored below `/var/lib/ceph/mon/ceph-a/`.

Before importing the monmap, let's have a look at the different rook
configurations that influence the ceph components

### Looking at the ConfigMap in detail: rook-ceph-mon-endpoints

As the name says, it contains the list of monitor endpoints:

kubectl -n rook-ceph edit configmap rook-ceph-mon-endpoints ...

csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["[2a0a:e5c0:0:15::fc2]:6789"... data: b=[2a0a:e5c0:0:15::9cd9]:6789,.... mapping: '{"node":{"a":{"Name":"server56","Hostname":"server56","Address":"2a0a:e5c0::...


As eventually we want the cluster and csi to use the in-cluster
monitors, we don't need to modify it right away.

### Looking at Secrets in detail: rook-ceph-admin-keyring

The first interesting secret is **rook-ceph-admin-keyring**, which
contains the admin keyring. The old one of course, so we can edit this
secret and replace it with the client.admin secret from our native
cluster.

We encode the original admin keyring using:

cat ceph.client.admin.keyring | base64 -w 0; echo ""


And then we update the secret it:

kubectl -n rook-ceph edit secret rook-ceph-admin-keyring


[done]

### Looking at Secrets in detail: rook-ceph-config

This secret contains two keys, **mon_host** and
**mon_initial_members**. The **mon_host** is a list of monitor
addresses. The **mon_host** only contains the monitor names, a, b, c, d and e.

The environment variable **ROOK_CEPH_MON_HOST** in the monitor
deployment is set to to **mon_host** key of that secret, so monitors
will read from it.



### Looking at Secrets in detail: rook-ceph-mon

This secret contains the following interesting keys:

* ceph-secret: the admin key (just the base64 key no section around
  it) [done]
* ceph-username: "client.admin"
* fsid: the ceph cluster fsid
* mon-secret: The key of the [mon.] section

It's important to mention to use `echo -n` when inserting
the keys or fsids.

[done]

### Looking at Secrets in detail: rook-ceph-mons-keyring

Contains the key "keyring" containing the [mon.] and [client.admin]
sections:

[mon.] key = ...

[client.admin] key = ... caps mds = "allow" caps mgr = "allow " caps mon = "allow " caps osd = "allow *"


Using `base64 -w0 <  ~/mon-and-client`.

[done]

### Importing the monmap

Getting the current monmap from the native cluster:

ceph mon getmap -o monmap-20220903

scp root@old-monitor:monmap-20220903


Adding it into the mon pod:

kubectl cp monmap-20220903 rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/tmp


Moving the old mon db away:

cd /var/lib/ceph/mon/ceph-a mkdir _old mv [a-z]* _old/


Recreating the mon fails, as the volume is mounted directly onto it:

% ceph-mon -i a --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key 2022-09-03 21:44:48.268 7f1a738f51c0 -1 '/var/lib/ceph/mon/ceph-a' already exists and is not empty: monitor may already exist

% mount | grep ceph-a /dev/sda1 on /var/lib/ceph/mon/ceph-a type ext4 (rw,relatime)


We can workaround this by creating all monitors on pods with other
names. So we can create mon b to e on the mon-a pod and mon-a on any
other pod.

On rook-ceph-mon-a:

for mon in b c d e; do ceph-mon -i $mon --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key; done


On rook-ceph-mon-b:

mon=a ceph-mon -i $mon --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key


Then we export the newly created mon dbs:

for mon in b c d e; do kubectl cp rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/var/lib/ceph/mon/ceph-$mon ceph-$mon; done

for mon in a; do kubectl cp rook-ceph/rook-ceph-mon-b-57d888dd9f-w8jkh:/var/lib/ceph/mon/ceph-$mon ceph-$mon; done


And finally we test it by importing the mondb to mon-a:

kubectl cp ceph-a rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/var/lib/ceph/mon/


And the other mons:

kubectl cp ceph-b rook-ceph/rook-ceph-mon-b-57d888dd9f-w8jkh:/var/lib/ceph/mon/


### Re-enabling the rook-operator

As the deployment

kubectl -n rook-ceph scale --replicas=1 deploy/rook-ceph-operator


Operator sees them running (with a shell)

2022-09-03 22:29:26.725915 I | op-mon: mons running: [d e a b c]


Triggering recreation:

% kubectl -n rook-ceph delete deployment rook-ceph-mon-a deployment.apps "rook-ceph-mon-a" deleted


Connected successfully to the cluster:

services: mon: 6 daemons, quorum red1,red2,red3,server4,server3,a (age 8s) mgr: red3(active, since 8h), standbys: red2, red1, server4 osd: 46 osds: 46 up, 46 in


A bit later:

mon: 8 daemons, quorum  (age 2w), out of quorum: red1, red2, red3, server4, server3, a, c,

d mgr: red3(active, since 8h), standbys: red2, red1, server4 osd: 46 osds: 46 up, 46 in


And a little bit later also the mgr joined the cluster:

services: mon: 8 daemons, quorum red2,red3,server4,server3,a,c,d,e (age 46s) mgr: red3(active, since 9h), standbys: red1, server4, a, red2 osd: 46 osds: 46 up, 46 in


And a few minutes later all mons joined successfully:

mon: 8 daemons, quorum red3,server4,server3,a,c,d,e,b (age 31s)
mgr: red3(active, since 105s), standbys: red1, server4, a, red2
osd: 46 osds: 46 up, 46 in


We also need to ensure the toolbox is being updated/recreated:

kubectl -n rook-ceph delete pods rook-ceph-tools-5cf88dd58f-fwwlc



### Original monitors vanish

Did not add bgp peering.
Cannot reach ceph through the routers.

Seems like rook did remove them.

Updating the ceph.conf for the native nodes:

mon host = rook-ceph-mon-a.rook-ceph.svc..,


### Post monitor migration issue 1: OSDs start crashing

A day after the monitor migration some OSDs start to crash. Checking
out the debug log we found the following error:

2022-09-05 10:24:02.881 7fe005ce7700 -1 Processor -- bind unable to bind to v2:[2a0a:e5c0::225:b3ff:fe20:3554]:7300/3712937 on any port in range 6800-7300: (99) Cannot assign requested address 2022-09-05 10:24:02.881 7fe005ce7700 -1 Processor -- bind was unable to bind. Trying again in 5 seconds 2022-09-05 10:24:07.897 7fe005ce7700 -1 Processor -- bind unable to bind to v2:[2a0a:e5c0::225:b3ff:fe20:3554]:7300/3712937 on any port in range 6800-7300: (99) Cannot assign requested address 2022-09-05 10:24:07.897 7fe005ce7700 -1 Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address 2022-09-05 10:24:07.897 7fe0127b1700 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-05 10:24:07.897 7fe0127b1700 -1 osd.49 100709 Got signal Interrupt 2022-09-05 10:24:07.897 7fe0127b1700 -1 osd.49 100709 Immediate shutdown (osd_fast_shutdown=true)


Trying to bind to an IPv6 address that is **not** on the system.

https://tracker.ceph.com/issues/24602

Calico/CNI does IP rewriting and thus tells the OSD the wrong IPv6
address.

Adding

public_addr = 2a0a:e5c0::92e2:baff:fe26:642c


to the node. Verifying the binding after restarting the crashing OSD:

[10:35:06] server4.place5:/var/log/ceph# netstat -lnpW | grep 3717792 tcp6 0 0 2a0a:e5c0::92e2:baff:fe26:642c:6821 ::: LISTEN 3717792/ceph-osd tcp6 0 0 :::6822 ::: LISTEN 3717792/ceph-osd tcp6 0 0 :::6823 ::: LISTEN 3717792/ceph-osd tcp6 0 0 2a0a:e5c0::92e2:baff:fe26:642c:6816 ::: LISTEN 3717792/ceph-osd tcp6 0 0 2a0a:e5c0::92e2:baff:fe26:642c:6817 ::: LISTEN 3717792/ceph-osd tcp6 0 0 :::6818 ::: LISTEN 3717792/ceph-osd tcp6 0 0 :::6819 ::: LISTEN 3717792/ceph-osd tcp6 0 0 2a0a:e5c0::92e2:baff:fe26:642c:6820 ::: LISTEN 3717792/ceph-osd unix 2 [ ACC ] STREAM LISTENING 16880318 3717792/ceph-osd /var/run/ceph/ceph-osd.49.asok


### Post monitor migration issue 1: OSDs start crashing

After roughly a week an OSD on the native cluster started to fail on
restart with the following error:

unable to parse addrs in 'rook-ceph-mon-a.rook-ceph.svc.p5-cow.k8s.ooo, rook-ceph-mon-b.rook-ceph.svc.p5-cow.k8s.ooo, rook-ceph-mon-c.rook-ceph.svc.p5-cow.k8s.ooo, rook-ceph-mon-d.rook-ceph.svc.p5-cow.k8s.ooo, rook-ceph-mon-e.rook-ceph.svc.p5-cow.k8s.ooo'


Checking the cluster, it seems rook has replaced mon-a with mon-f:

[22:38] blind:~% kubectl -n rook-ceph get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 2a0a:e5c0:0:15::f2ac 8080/TCP,8081/TCP 7d5h csi-rbdplugin-metrics ClusterIP 2a0a:e5c0:0:15::5fc2 8080/TCP,8081/TCP 7d5h rook-ceph-mgr ClusterIP 2a0a:e5c0:0:15::c31c 9283/TCP 7d5h rook-ceph-mon-b ClusterIP 2a0a:e5c0:0:15::9cd9 6789/TCP,3300/TCP 7d5h rook-ceph-mon-c ClusterIP 2a0a:e5c0:0:15::fc2 6789/TCP,3300/TCP 7d5h rook-ceph-mon-d ClusterIP 2a0a:e5c0:0:15::b029 6789/TCP,3300/TCP 7d5h rook-ceph-mon-e ClusterIP 2a0a:e5c0:0:15::8c86 6789/TCP,3300/TCP 7d5h rook-ceph-mon-f ClusterIP 2a0a:e5c0:0:15::2833 6789/TCP,3300/TCP 3d13h


At this moment it is unclear why ceph does it, but if the native hosts
had already been migrated, this would probably not have caused an
issue. However as long as ceph.conf files are deployed with static
references to the monitors, this problem might repeat.


## Changelog

### 2023-05-18

* rook 1.7.11 does not work on kubernetes 1.26.1 anymore
* PodSecurityPolicy is missing

The Kubernetes API could not find policy/PodSecurityPolicy for requested resource rook-ceph/00-rook-ceph-operator. Make sure the "PodSecurityPolicy" CRD is installed on the destination cluster.


Same issue on rook 1.8.10:

      kind: PodSecurityPolicy
      message: >-
        The Kubernetes API could not find policy/PodSecurityPolicy for
        requested resource rook-ceph/00-rook-privileged. Make sure the
        "PodSecurityPolicy" CRD is installed on the destination cluster.
      name: 00-rook-privileged
      namespace: rook-ceph
      status: SyncFailed
      syncPhase: Sync
      version: v1beta1
  revision: v1.8.10
  source:


* rook 1.9.12 deploys in kubernetes 1.26.1

2023-05-18 11:44:36.169248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed the ceph version check: the version does not meet the minimum version "15.2.0-0 octopus" ```

requires 15.2.0 octopus as a minimum

2022-09-10

Added missing monitor description

2022-09-03

Next try starting for migration
Looking deeper into configurations

2022-08-29

Added kubernetes/kubeadm bootstrap issue

2022-08-27

The initial release of this blog article
Added k8s bootstrapping guide

Follow up or questions

You can join the discussion in the matrix room #kubernetes:ungleich.ch about this migration. If don't have a matrix account you can join using our chat on https://chat.with.ungleich.ch.

[matrix] Conduit beta at ungleich

2022-07-10T00:00:00Z

News from the Matrix Home Server department

Here at ungleich we always have a look around, which Matrix home server to use. Obviously, you all know about synapse, the reference server. Some of you might also know about dendrite, an experimental, maybe at some point in the future replacement of synapse.

But did you know about conduit, a tiny Matrix homeserver written in Rust? Well, now you do.

Next generation Matrix Home Server?

At ungleich we like to see what is coming next and conduit certainly has the potential to become an interesting, production grade homeserver. At the moment there are some issues open for the 1.0 milestone, however conduit is actually already quite usable.

How stable you ask? Well, ...

Phasing in public beta server at ungleich

To give you a preview, we have setup an IPv6 only instance of conduit at conduit.ungleich.ch.

This server is open for public use, anyone (well, anyone with IPv6...) can register. However note: this is only a test instance. We might at any point in the future delete, replace or reset it. It is not backed and not a production service. It's main intent is to allow you to easily test out Conduit.

If you are looking for a production grade homeserver for yourself, have a look at our hosted Matrix offerings.

That said - give it a shot. We think Conduit is a promising homeserver, especially given its focus on low resource usage.

... why IPv6 only?

There are several reason for it:

IPv6 only is much easier to setup than IPv6+IPv4
This is a public beta service for which we try to keep the effort low
It's good test for us to see if conduit works in our IPv6 only networks
Global IPv6 traffic is nearing 50% and many countries already exceed 50%

If you cannot access the conduit server because you don't have IPv6, you can always get IPv6 via the IPv6VPN.

Follow up or questions

You can join the discussion either on the conduit server or you can also join our chat on https://chat.with.ungleich.ch (IPv6+IPv4 Synapse based Matrix server).

If you have anything to share about conduit, feel free to join the room #conduit:fachschaften.org.

Let's get started with Kubernetes

2022-06-03T00:00:00Z

It's time for Kubernetes

Like everybody else, you might have been following the topic of kubernetes for a while and maybe you wondered whether or not it's about time to learn kubernetes. So have we at ungleich. Over the years we tested kubernetes many times and quite often had the verdict "no, not yet".

About one and a half year ago, our verdict finally changed. After much testings and learnings, we came to the conclusion that Kubernetes is stable enough to be running in the ungleich environment. Speaking of our environment, it is not only IPv6-only, but also needs to support a huge variety of workloads.

Open Source and Open Teaching

Applying Kubernetes on our own datacenter required a lot of intense learning and real-life problem solving. We won't say that was always easy, but it has been quite rewarding. One of the main takeaways is that it can be tough to kickstart Kubernetes. Since it has a lot of different use-cases, concepts and configurations, it can be easily overwhelming.

At ungleich we have always been standing for Open Source and Open Communication and we want to continue this by passing on our knowledge about Kubernetes.

Is Kubernetes for me?

That depends and there are many factors to consider. If you are an organisation and you want to standardise your application deployments or increase the efficiency of deployments, Kubernetes is a good choice.

If you are an engineer or a developer and you are considering improving your career, learning Kubernetes is a good choice.

2 Beginner Courses starting in Digital Glarus

For this reason we will start 2 beginner courses this June (Update: only 2 places left) in the beautiful Hacking Villa in Diesbach, Switzerland.

Setting up Kubernetes

The first course, Kubernetes Setup Course focuses on how to set up Kubernetes clusters. It is intended for Sysadmins and DevOps engineers that want to operate Kubernetes clusters.

Using Kubernetes

After having a cluster setup, the big question is how to actually use it. Our second course, the Kubernetes Usage Course is focussing on the aspect of understanding the concepts of Kubernetes.

On-Site and Online

Both courses are available On-Site in Switzerland as well as Online (choose the "Learning only" package for this one). If you choose to be onsite, you will visit us in the beautiful Swiss Alps and lodge in a historic, protected building.

Intense + Personal + Relaxing

At ungleich we know that intense learning (6 hours per day, 3 hours in the morning, 3 hours in the afternoon) also requires fitting calming down to be able to concentrate.

For this reason we have local recommendations for hiking paths, swimming or - if you join the winter course - skiing.

Our experience has shown that learnings in small groups with personal training are most effective. For this reason the size of the training groups is limited to 6 people.

A or B or A+B?

If you are considering participating in the setting up or using Kubernetes course: you can take either of them or both of them, they work independently of each other.

Note: for both courses we have 2 places free - if you are interested, you can apply directly on the course websites:

Looking forward to seeing you

Ever since we started our company ungleich we have done things a bit differently, focusing on sustainability and open source. Since the launch of project Digital Glarus in 2015, many people from all over the world have visited us, learned with us and a lot of them even stayed in our community via our Open Chat. We look forward to welcoming you to Digital Glarus or teaching you online and maybe staying with us even longer.

Our Open Chat is based on Matrix and if you are curious about what we do and write about Kubernetes, feel free to join

kubernetes:ungleich.ch.

Hosting Django Apps in Kubernetes [WIP]

2022-06-03T00:00:00Z

Situation

At ungleich we are hosting quite a lot of Django applications. As of 2022-06-03, most of them are still deployed on a traditional VM based setup.

We are using this blog entry to document a possible blueprint and the progress of migration at ungleich.

General design

Our Kubernetes clusters usually use ArgoCD for deployments, so Django applications should potentially be defined the same way.

Most of our kubernetes applications are defined in helm charts and thus the "general django application" should probably also be defined in a helm chart.

Freedom of choice

While as a hoster it might be tempting to define a specific image that Django applications should be using (like python), but we want to give us and our customers the freedom to choose the image they use themselves. It might potentially even come from a private registry.

Interface definition

All of our Django applications are using Postgresql for storing data. Postgresql is used by quite some other applications that we deployed in k8s, so this is a no-brainer. Django hosting at ungleich, even in k8s, will be based on Postgresql.

Static data of Django applications can easily be stored on a PVC. This has the drawback that filesystem PVCs based on ceph block devices are usually RWO and thus in case of restart, there will be a short downtime.

This is, generally speaking probably accepted, like a deploy would have caused a short downtime on a VM as well.

However alternatives would be a shared filesystem (such as NFS/CephFS), but they are usually slower than dedicated block devices - so reliability can be traded against speed. Maybe we offer both options or add an NFS server as an option to our Django Hosting.

Django startup / processes

On startup, Django will need to ensure the database schema has been upgraded to the latest version. so something like python manage.py migrate should probably be called in an InitContainer for most apps. We could specify that the customer provided container supports multiple commands:

/init - anything that needs to be done once on startup
/run - something that runs the actual site

Some django apps however utilise Celery or Django Q for async tasks. We don't know which system is used, but it would be easy to add a flag to the hosting whether or not a third container should be utilised. Thus we could define:

If async container is defined, enable it and run /async (or similar)

Secrets

Django applications usually have some kind of secrets and most Django applications have DIFFERENT types of secrets. Thus defining a specific environment variable does not seem to be a smart idea.

Instead, we should probably offer to store secrets in something like SealedSecrets.

Database connection information is provided by default and is cluster/app specific.

Deployment

We are likely going with a git-ops style deployment in which everything is defined for the Django app. This repository is read write for each client/customer.

This probably includes:

(Possible encrypted) Secrets
Image definitions
Maybe even (part of) a pod definition?

Some parameters are likely to be stored in a different, ungleich only writable repository, such as:

Size of RAM
Number of CPUs
Storage size (Postgresql, Static files, ...)

Status

This document is still WIP and will be used as a basis for deploying our own Django apps first.

URGENT: Infrastructure update Announcement

2022-04-01T00:00:00Z

APRIL's FOOLS DAY

None of the news below is true - this was just part of our 2022 April's fools day contribution.

Urgent release: ungleich infrastructure update

To the attention to every ungleich customer: today we are updating our infrastructure to the latest technologies. All changes will be applied at 23:42 CEST today.

What does that mean to you? The following sections answer this in detail.

Network upgrades: Turning IPv4 off

As of today all IPv4 traffic will be disabled. IPv4 has become too expensive and it is not worth keeping it online.

The migration will be as follows: every IPv4 address like A.B.C.D will be translated to an IPv6 address like 2a0a:e5c0:b00::A.B.C.D.

As a bonus, every customer will get their own "cafe" IPv6 network.

Additionally every server will be connected also via SCION, the new Internet.

Network upgrade: Removal of old connections: no more 2G/3G/4G

You may be aware that slow Internet connections are draining server resources: long lasting connections without much data transfer is a burden for every server. For that reason, we will disable access to our data center from any 2G/3G/4G mobile connection.

You are required to use 5G, 6G, 7G, Fiber or 56k modems to access Data Center Light. The latter was included as a fallback, in case you don't have support for a modern Internet connection.

Storage migration

Our storage is so far based on Ceph, a distributed storage. To follow modern standards, we are migrating all our data to the blockchain.

To avoid vendor logins and possible shortcomes of a particular blockchain, we are introducting the "ungleich blockchain", which stores data on all available blockhains.

If you are worried about privacy implications of all data being public, don't worry: we are encrypting all data using an Enigma machine.

Optionally, on request, we can also encrypt your data as NFTs.

In regards to performance, we expect a huge performance increase, as all data will be saved everywhere.

Computing power

As you might have read recently, we are investigating heating houses with the server heat. This project is now in productive state and we require much more computing power.

For that reason, at the same time today, our whole server infrastructure will be replaced with quantum computers.

As qubits, the equivalent of bits on quantum computers, operate on probabilities, all calculations for our customers are not guaranteed anymore, but instead they will be executed by a certain probability.

The exact probability is derived from your society behaviour points. If you are in the good place you get an additional uncertainty factor of 3.14.

Removal of Kubernetes

We recently introduced kubernetes in our infrastructure. After only 2 years of R&D we will have to shutdown all Kubernetes based clusters. The main reason is that the actual complexity of Kubernetes systems is not high enough. Plainly speaking, our engineers are bored. For that reason:

We will replace Kubernetes with a home made solution that consists solely of a manually managed cluster of Windows Servers. Microsoft is a direct partner in this and provides us with a new Windows version: Microsoft Windows Quantum ME 3.11. We expect great stability from this release with a minimum complexity factor of 1000x added.

Maintenance and Customer support

We have always been very community orientated. Today we wanted to take the next big step and we turn over the customer support and system maintenance to the community. What does that mean specifically?

Anyone interested in maintaining our infrastructure can get full root access on all systems.

As everyone now has full root access, the customer support is also shifted to a decentralised, bittorrent based support community.

If you have any questions to the recent changes, please connect the decentrally organised support community. Unfortunately, due do the decentralisation, we cannot offer a single contact reference.

However the Web3 advocate of your choice will certainly be able to provide you answers.

100: the number of servers at ungleich

2022-02-14T00:00:00Z

The day of celebration, 2022-02-14

Today is the day that we put server100 online in our data center. Server100. Who would have thought that when we started our journey with the Data Center Light around 2017?

Just on this Valentine's day, we have actually received 2 new servers to support the growing storage needs of our customers, so server100 is celebrating in the data center with server101.

Big data

You might remember this slogan from the 2017's, "Big data" or "what can you do if you have a lot of data". We see big data a bit differently. We don't ever analyse our customer data or sell it to third parties, because we know that for every of our customers privacy is an important.

No, at ungleich big data means "storing a lot of data". And that is what server100 and server101 are going to do: each of them will provide around 400 Terabyte of storage.

This is a bit of a special case in our infrastructure, as these servers are dedicated to specific customers. Usually, all of our storage is provided by our Ceph clusters, but in this case these machines are providing independent storage.

Kubernetes + IPv6 everywhere

They are however integrated into our kubernetes clusters so that the actual workload is scheduled via kubernetes to these specific hosts.

As usual, the servers are running in IPv6 only networks, but have access to the Internet via NAT64.

More about server100

So, what kind of server is this server100 anyway? It is a 24 bay, Supermicro X9QRi-F with 64 cores and 512GB RAM. It is connected to our network with 2x 10 Gbit/s network cards running LACP bonding. Not only in terms of computing it is big, it is with it's 1620W PSU also one of the "heaviest" servers in our data center (the average PSU is rated somewhere in the 1000W area).

Some more insights coming from the terminal:

[17:33] server100.place10:~# cat /proc/cpuinfo  | grep ^process | wc -l
64
[17:33] server100.place10:~# free -g
              total        used        free      shared  buff/cache   available
Mem:            503           0         502           0           1         501
Swap:             0           0           0

Obviously, the uptime is not yet that high...

[17:46] server100.place10:~# uptime
 17:46:42 up  1:37,  load average: 0.00, 0.00, 0.00

... neither are the measured temperatures:

root@2157f4626763:/# ipmitool sensor  | grep degrees
CPU1 Temp        | 27.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 97.000    | 100.000   | 102.000
CPU2 Temp        | 28.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 97.000    | 100.000   | 102.000
CPU3 Temp        | 28.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 97.000    | 100.000   | 102.000
CPU4 Temp        | 27.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 97.000    | 100.000   | 102.000
PCH Temp         | 43.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000
System Temp      | 16.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Rear Left Temp   | 17.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Rear Right Temp  | 14.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
root@2157f4626763:/#

With the addition of server100 and server101 we are about to crack the 10 TiB RAM barrier. Prior to the deployment of these two, a total of 8 TiB had been deployed in the Data Center Light.

As you can see, we have a lot of fun with our latest servers. And this brings us to the important point: celebration.

Celebration

Over the years we learned many great stories and have done many great projects together with our customers and partners. Many of them have turned into friends and we know the spirit of each and every project that came to us.

In this sense we want to celebrate reaching that big number with you and will give a 100% discount on any hosting order of 100 CHF or more for the first month.

Checkout the details on the 100 servers - 100 percent discount page. And now, happy 100 everyone!

If you feel like celebrating, you can also join us on our open chat.

IPv6 addresses for free

2021-12-23T00:00:00Z

TL;DR

Are you, your organisation or someone you know interested in public, static, addressable IPv6 address space? But have you refrained from getting it because of cost issues? And/or would you be interested in affordable (free/almost free) IPv6 address space?

If yes, then please reach out to us via:

With this information:

Who are you?
What do you need or plan to use the IPv6 space for?
Are we allowed to share your information publicly?
What are your financial constraints?

The last question is mainly to find out why you would not apply to become an LIR at one of the RIRs. However if you don't want to share, the last question can be skipped.

Note:

This is not about routing or connectivity, but merely about assignment of address space.

The background: free = ULA

At the moment the only "free" available IPv6 address space is ULA space. However there is no official ULA registry, even though we provide a ULA registry at ungleich. We are not the only ones though, DN42 also registers ULA space.

ULA space is not only not officially managed, but also comes with the drawback that it will never be routed through the Internet.

Global Unique Addresses (GUA) are paid

Global Unique Addresses, even if not routed or announced in the Internet, are something you can get from an RIR, if you are a registered LIR. Being an LIR costs 1400 Euro per year + a one time fee.

One theory is that every organisation can afford this, but at ungleich we imagine this is stopping organisations from pursuing GUA.

This survey

This is why we started this page, to find out which organisations are looking for IPv6 address space or are using ULA because of cost issues. Answering this survey helps to find out whether or not there should be afford to either

Establish an official ULA registry
To provide GUA address space for free

So please spread the word - the more answers there are, the easier it will be to continue the discussion. We plan to publish all answers that are allowed to be shared publicly in a git repository.

Running Workadventure in Kubernetes IPv6 only

2021-12-19T00:00:00Z

Overview

At ungleich we are often running software in IPv6 only environments and recently even more in IPv6 only Kubernetes clusters. Today we had a look at running WorkAdventure in an IPv6 only kubernetes cluster.

Status: waiting for bugfix

At the moment it looks like as if WorkAdventure cannot run in IPv6 only Kubernetes clusters. The frontend displays the infamous "Network Error" messages. When checking the backend, it displays an error that it cannot resolve the redis hostname, which seems to be a bug in the resolver code, as the hostname does resolve, albeit only to an IPv6 address.

The code

As usual you can find our code in the ungleich-k8s repository, which contains development iterations at the moment:

v1: initial conversions from docker-compose using kompose
v2: Adjust manifests so that pods generally speaking run
v3: Turned into a helm chart with most services running stable

If you want to give it a spin yourself, here is how to:

Setup an IPv6 only kubernetes cluster
Ensure you have helm locally installed
Ensure you have
Clone the ungleich-k8s repo (see link above)
cd apps/workadventure/

And use

helm upgrade --install workadventure v3/

Next steps

We are currently waiting to hear back from the redis bug report and will continue developing after the backend is running stable in an IPv6 only environment.

If you are interested in this development, feel free to join us on the open kubernetes chat.

Running Django in Kubernetes IPv6 only

2021-12-19T00:00:00Z

Overview

At ungleich we run quite some amount of Django applications, some for ourselves, some on behalf of our customers. Most of them are currently running "natively" on virtual machines. Our objectives are to

Slowly move our own workload into IPv6 only kubernetes clusters
Slowly move our customer applications into IPv6 only kubernetes clusters
Offer modern Django Hosting based on Kubernetes

Status 2021-12-19: initial design phase

At the moment we are looking into what we require for running Django applications inside Kubernetes. Some parts are still open, some are already set:

PostgreSQL as the database: this is our standard so far and we don't plan to change it

Undefined at the moment:

Where/how to generate and serve static files
- Same container?
- Separate container?
Backup sidecare
- Restic?
- pg_dump?
- media / upload files
App server
- uwsgi
- (g)unicorn

One container providing port 80

The outside world does not need to know what is running inside
Thus the django container should:
- Expose port 80
- Serve staticfiles on its own
One can use any of the wsgi/asgi servers
- Will provide example for some of them

Follow up

If you are interested in Kubernetes or Django, feel free to join us on our open chat.

Public release of the ungleich container registries

2021-12-11T00:00:00Z

TL;DR

Today we opened up two public container registries from ungleich. You can use them to pull container images released by ungleich or to utilise the container registry caches.

Overview

You can pull public ungleich container releases from https://harbor.ungleich.svc.p10.k8s.ooo/ungleich-public
You can use the hub.docker.com cache on https://harbor.ungleich.svc.p10.k8s.ooo/dockerhub
You can use the quay.io cache on https://harbor.ungleich.svc.p10.k8s.ooo/quayio
You can use the hub.docker.com cache on https://harbor.ungleich.svc.c2.k8s.ooo/dockerhub (development cluster)
You can use the quay.io cache on https://harbor.ungleich.svc.c2.k8s.ooo/quayio (development cluster)

All registries are IPv6 only and the caches are intended to be used by Data Center Light customers to avoid the IPv4 throttling that one encounters when pulling images from IPv6 only VM.

You can test the ungleich-public repository as follows:

docker pull harbor.ungleich.svc.p10.k8s.ooo/ungleich-public/ungleich-certbot:0.3.3

Usage

If you are using cri-o you can use a registries.conf as follows:

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "harbor.ungleich.svc.p10.k8s.ooo/dockerhub"

[[registry]]
prefix = "quay.io"
location = "harbor.ungleich.svc.p10.k8s.ooo/quayio"

[[registry.mirror]]
location = "harbor.ungleich.svc.c2.k8s.ooo/dockerhub"

[[registry.mirror]]
location = "harbor.ungleich.svc.c2.k8s.ooo/quayio"

Next steps and more of it

More caches are to follow and we will add a list of publicly available images in the future. We do not plan to make the registries IPv4 accessible, as all workload inside Data Center Light is IPv6 reachable and we think there is no need for IPv4 connectivity anymore. You can read more about the ungleich kubernetes infrastructure or join the #kubernetes:ungleich.ch Matrix chat.

IPv6, VPN and DNS entries

2021-10-13T00:00:00Z

TL; DR

With IPv6, DNS management of protected networks can be simplified. IPv6 VPNs can use simplified DNS configurations to simplify the network configurations by just using public, restricted DNS entries.

VPN and DNS in the IPv4 world

VPNs in the IPv4 world are often used to create site-to-site tunnels, allowing different networks to talk to each other. A typical case is that organisation A needs to access protected resources of organisation B and maybe even vice-versa. So a typical VPN looks like this:

Organisation A
-------------

Protected Host A ---------- Router/VPN gateway
(10.0.0.42/24)                      |
                                    |
                                    |
Organisation B                  (Internet)
--------------                      |
                                    |
                                    |
Protected Host B ---------- Router/VPN gateway
(10.20.0.42/24)
Host name: lakeside.int.org-b.example.com

Now if the Protected Host A and Protected Host B want to communicate with each other on IP basis, this is no problem (I am not elaborating on the problems of IP collisions in this article, a follow up article will follow soon).

However if Protected Host A wants to reach the Protected Host B via its internal DNS name lakeside.int.org-b.example.com, this is usually a problem, for multiple reasons:

Protected Host A might not know the right internal DNS server to query for int.org-b.example.com.
Protected Host A might know the right internal DNS server to query for int.org-b.example.com, but might not have access to it via the VPN
The DNS records for int.org-b.example.com often are intentionally not published to public DNS for multiple reasons: privacy related or because administrators don't like to publish RFC1918 records into public DNS records

VPN and DNS in the IPv6 world

There are multiple ways of how VPNs can be built in the IPv6 world, including usage of the private IPv4 addresses equivalent named Unique Local Address (ULA). However instead of using ULA, I will today show an approach that is more "IPv6 native", using Global Unique Addresses (GUA), or what is simply known as "public IPv6 address".

While you might have heard it, I will repeat nonetheless: there are enough IPv6 addresses for every practical use case that we imagine at the moment. This is important, because we can use globally unique IPv6 addresses inside the VPN.

Isn't that a problem? Publicly reachable IPv6 addresses inside a VPN? It would, if the addresses were globally reachable. In the IPv6 world nothing speaks against having globally unique, but non-routed IPv6 addresses. This is actually a perfect match and much better than we can do in the IPv4 world:

Both organisations A and B can acquire globally unique addresses. Let's say they organisation A acquires 2001:db8:0::/48 and organisation B acquires 2001:db8:1::/48.
Both organisations have two options: they can announce their IPv6 range to the Internet and block access to their internal network or
both they can even consider not to announce their network at all (there is not route in the Internet for it)

In either case, both organisations will usually select a sub network of size /64 for the resources they want to expose via the VPN. Let's say organisation A chooses 2001:db8:0:cafe::/64 and organisation B chooses 2001:db8:1:7ea::/64. Putting this in context, their VPN now looks like this:

Organisation A
-------------

Protected Host A ---------- Router/VPN gateway
(2001:db8:0:cafe::42/64)            |
                                    |
                                    |
Organisation B                  (Internet)
--------------                      |
                                    |
                                    |
Protected Host B ---------- Router/VPN gateway
(2001:db8:1:7ea::42/64)            |
Host name: lakeside.int.org-b.example.com

Now, how does this change the DNS server situation? Because we are using IPv6, we have many more options:

a) We can publish the DNS records of the domain int.org-b.example.com globally. While access to the network 2001:db8:1:7ea::/64 is only possible via VPN, nothing speaks against having the records in a public DNS server. However, some administrators advocate to not publish them publicly for privacy reasons. That is the same logic as publishing or not publish the RFC1918 (10.x.y.z) addresses in the IPv4 world.
b) We can publicly/globally delegate the domain int.org-b.example.com to a nameserver that is only reachable via the VPN.
c) We can proceed the same as in the IPv4 world and have a disconnect, internal DNS server that is responsible for int.org-b.example.com.

Option (a) is often seen as a security risk and it can be debated whether someone who can already guess the correct hostname and retrieve it's IP address is really a significant higher security thread than anybody just guessing IP addresses.

Option (c) is the typical case for IPv4 based VPNs and is causing above illustrated issues.

Option (b) is the one that makes IPv6 VPNs much more interesting than IPv4 based VPNs:

The world can know that there is an internal domain int.org-b.example.com and find out which DNS servers are responsible for it.
However an attacker easily guesses that internal networks exist anyway.

Let's have a look at sample nameserver entries in detail:

int.org-b.example.com. NS ns-int1.org-b.example.com.
int.org-b.example.com. NS ns-int2.org-b.example.com.

What does that mean? Anyone in the world can retrieve the information that int.org-b.example.com has two DNS servers. However the DNS servers responsible for org-b.example.com can hide the IP addresses of ns-int1.org-b.example.com and ns-int2.org-b.example.com for everyone, but hosts coming from organisation A. Or even if the IP addressses of ns-int1.org-b.example.com and ns-int2.org-b.example.com are world known, access to them can easily be prevented.

The measures for this can for instance be DNS views or firewall entries. In practice this means for VPNs in the IPv6 world:

Organisation A
-------------

Protected Host A: what is the IP address of lakeside.int.org-b.example.com?
DNS Server of Organisation B: 2001:db8:1:7ea::42


Outside party
------------
Outside Hosts: what is the IP address of lakeside.int.org-b.example.com?

a) DNS Server of Organisation B: there is no domain
   int.org-b.example.com (DNS view restriction)
b) DNS Server of Organisation B: these are the nameserver for
   int.org-b.example.com, but you cannot reach them (firewall protection)

Summary

For IPv6 based VPNs you can get away without reconfiguring your source networks for DNS servers of the destination party. The target party always needs to ensure proper access control to internal resources, so there is no additional overhead.

DNS, correctly used in the IPv6 VPN world, is a really smooth operation. This is why we recommend to use IPv6 as a basis for VPNs.

Bye, bye netboot

2021-08-31T00:00:00Z

Introduction

Since the very beginning of the Data Center Light project our servers have been somewhat stateless and booted from their operating system from the network.

From today on this changes and our servers are switched to boot from an disk (SSD/NVMe/HDD). While this first seems counter intuitive with growing a data center, let us explain why this makes sense for us.

Netboot in a nutshell

There are different variants of how to netboot a server. In either case, the server loads an executable from the network, typically via TFTP or HTTP and then hands over execution to it.

The first option is to load the kernel and then later switch to an NFS based filesystem. If the filesystem is read write, you usually need one location per server or you mount it read only and possibly apply an overlay for runtime configuration.

The second option is to load the kernel and an initramfs into memory and stay inside the initramfs. The advantage of this approach is that no NFS server is needed, but the whole operating system is inside the memory.

The second option is what we used in Data Center Light for the last couple of years.

Netboot history at Data Center Light

Originally all our servers started with IPv4 PXE based netboot. However as our data center is generally speaking IPv6 only, the IPv4 DHCP+TFTP combination is an extra maintenance and also a hindrance for network debugging: if you are in a single stack IPv6 only network, things are much easier to debug. No need to look for two routing tables, no need to work around DHCP settings that might interfere with what one wants to achieve via IPv6.

As the IPv4 addresses became more of a technical debt in our infrastructure, we started flashing our network cards with ipxe, which allows even older network cards to boot in IPv6 only networks.

Also in an IPv6 only netboot environment, it is easier to run active-active routers, as hosts are not assigned DHCP leases. They assign addresses themselves, which scales much nicer.

Migrating away from netbooting

So why are we migrating away from netbooting, even after we migrated to IPv6 only networking? There are multiple aspects:

On power failure, netbooted hosts lose their state. The operating system that is loaded is the same for every server and needs some configuration post-boot. We have solved this using cdist, however the authentication-trigger mechanism is non-trivial, if you want to keep your netboot images and build steps public.

The second reason is state synchronisation: as we are having multiple boot servers, we need to maintain the same state on multiple machines. That is solvable via CI/CD pipelines, however the level of automation on build servers is rather low, because the amount of OS changes are low.

The third and main point is our ongoing migration towards kubernetes. Originally our servers would boot up, get configured for providing ceph storage or to be a virtualisation host. The amount of binaries to keep in our in-memory image was tiny, in the best case around 150MB. With the migration towards kubernetes, every node is downloading the containers, which can be comparable huge (gigabytes of data). The additional pivot_root workarounds that are required for initramfs usage are just an additional minor point that made us question our current setup.

Automating disk based boot

We have servers from a variety of brands and each of them comes with a variety of disk controllers: from simple pass-through SATA controllers to full fledged hardware raid with onboard cache and battery for protecting the cache - everything is in the mix.

So it is not easily possible to generate a stack of disks somewhere and then add them, as the disk controller might add some (RAID0) meta data to it.

To work around this problem, we insert the disk that is becoming the boot disk in the future into the netbooted servers, install the operating system from the running environment and at the next maintenance window ensure that the server is actually booting from it.

If you are curious on how this works, you can checkout the script that we use for Devuan/Debian and Alpine Linux

The road continues

While a data center needs to be stable, it also needs to adapt to newer technologies or different flows. The disk based boot is our current solution for our path towards kubernetes migration, but who knows - in the future things might look different again.

If you want to join the discussion, we have a Hacking and Learning (#hacking-and-learning:ungleich.ch) channel on Matrix for an open exchange.

Oh and in case you were wondering what we did today, we switched to disk based booting - that case is full of SSDs, not 1'000 CHF banknotes.

Configuring bind to only forward DNS to a specific zone

2021-07-25T00:00:00Z

Introduction

In this article we'll show you an easy solution to host DNS zones on IPv6 only or private DNS servers. The method we use here is DNS forwarding as offered in ISC BIND, but one could also see this as DNS proxying.

Background

Sometimes you might have a DNS server that is authoritative for DNS data, but is not reachable for all clients. This might be the case for instance, if

your DNS server is IPv6 only: it won't be directly reachable from the IPv4 Internet
your DNS server is running in a private network, either IPv4 or IPv6

In both cases, you need something that is publicly reachable, to enable clients to access the zone, like show in the following picture:

The problem: Forwarding requires recursive queries

ISC Bind allows to forward queries to another name server. However to do so, it need to be configured to allow handling recursive querying. However, if we allow recursive querying by any client, we basically create an Open DNS resolver, which can be quite dangerous.

The solution

ISC Bind by default has a root hints file compiled in, which allows it to function as a resolver without any additional configuration files. That is great, but not if you want to prevent it to work as forwarder as described above. But we can easily fix that problem. Now, let's have a look at a real world use case, step-by-step:

Step 1: Global options

In the first step, we need to set the global to allow recursion from anyone, as follows:

options {
    directory "/var/cache/bind";

    listen-on-v6 { any; };

    allow-recursion { ::/0; 0.0.0.0/0; };
};

However as mentioned above, this would create an open resolver. To prevent this, let's disable the root hints:

Step 2: Disable root hints

The root hints are served in the root zone, also know as ".". To disable it, we give bind an empty file to use:

zone "." {
        type hint;
        file "/dev/null";
};

Note: in case you do want to allow recursive function for some clients, you can create multiple DNS views.

Step 3: The actual DNS file

In our case, we have a lot of IPv6 only kubernetes clusters, which are named xx.k8s.ooo and have a world wide rachable CoreDNS server built in. In this case, we want to allow the domain c1.k8s.ooo to be world reachable, so we configure the dual stack server as follows:

zone "c1.k8s.ooo"  {
   type forward;
   forward only;
   forwarders { 2a0a:e5c0:2:f::a; };
};

Step 4: adjusting the zone file

In case you are running an IPv6 only server, you need to configure the upstream DNS server. In our case this looks as follows:

; The domain: c1.k8s.ooo
c1                          NS      kube-dns.kube-system.svc.c1

; The IPv6 only DNS server
kube-dns.kube-system.svc.c1 AAAA    2a0a:e5c0:2:f::a

; The forwarding IPv4 server
kube-dns.kube-system.svc.c1 A       194.5.220.43

DNS, IPv6, Kubernetes?

If you are curious to learn more about either of these topics, feel free to join us on our chat.

GLAMP #1 2021

2021-07-17T00:00:00Z

Glamp 2021 has been CANCELLED

Due to the rising number of Coronavirus infections, travel restrictions and travel uncertainties, we have decided to CANCEL the GLAMP2021.

Tl;DR

Get your tent, connect it to power and 10Gbit/s Internet in the midst of the Glarner mountains. Happenening Thursday 2021-08-19 to Sunday 2021-08-22. Apply for participation by mail (information at the bottom of the page).

Introduction

It has been some time since our last Hack4Glarus and we have been missing all our friends, hackers and participants. At ungleich we have been watching the development of the Coronavirus world wide and as you might know, we have decided against a Hack4Glarus for this summer, as the Hack4Glarus has been an indoor event so far.

No Hack4Glarus = GLAMP

However, we want to try a different format that ensures proper safety. Instead of an indoor Hack4Glarus in Linthal, we introduce the Glarus Camp (or GLAMP in short) to you. An outdoor event with sufficient space for distancing. As a camping site we can use the surrounding of the Hacking Villa, supported by the Hacking Villa facilities.

Compared to the Hack4Glarus, the GLAMP will focus more on relaxation, hangout than being a hackathon. We think times are hard enough to give everyone a break.

The setting

Many of you know the Hacking Villa in Diesbach already. Located just next to the pretty waterfall and the amazing Legler Areal. The villa is connected with 10 Gbit/s to the Data Center Light and offers a lot of fun things to do.

Coronavirus measures beforehand

To ensure safety for everyone, we ask everyone attending to provide a reasonable proof of not spreading the corona virus with one of the following proofs:

You have been vaccinated
You had the corona virus and you are symptom free for at least 14 days
You have been tested with a PCR test (7 days old at maximum) and the result was negative

All participants will be required to take an short antigen test on site.

Please do not attend if you feel sick for the safety of everyone else.

Coronavirus measures on site

To keep the space safe on site as well, we ask you to follow these rules:

Sleep in your own tent
Wear masks inside the Hacking Villa
- Especially if you are preparing food shared with others
Keep distance and respect others safety wishes

Hacking Villa Facilities

Fast Internet (what do you need more?)
A shared, open area outside for hacking
Toilets and bath room located inside

What to bring

A tent + sleeping equipment
Fun stuff
- Your computer
- Wifi / IoT / Hacking things
If you want wired Internet in your tent: a 15m+ Ethernet cable
- WiFi will be provided everywhere

What is provided

Breakfast every morning
A place for a tent
Power to the tent (Swiss plug)
WiFi to the tent
Traditional closing event spaghetti

What you can find nearby

A nearby supermarket (2km) reachable by foot, scooter, bike
A waterfall + barbecue place (~400m)
Daily attractions such as hacking, hiking, biking, hanging out

Registration

As the space is limited, we can accomodate about 10 tents (roughly 23 people). To register, send an email to support@ungleich.ch based on the following template:

Subject: GLAMP#1 2021

For each person with you (including yourself):

    Non Coronavirus proof:
    (see requirements on the glamp page)

    Name(s):
    (how you want to be called)

    Interests:
    (will be shown to others at the glamp)

    Skills:
    (will be shown to others at the glamp)

    Food interests:
    (we use this for pooling food orders)

    What I would like to do:
    (will be shown to others at the glamp)

The particaption fee is 70 CHF/person (to be paid on arrival).

Time, Date and Location

Arrival possible from Wednesday 2021-08-18 16:00
GLAMP#1 starts officially on Thursday 2021-08-19, 1000
GLAMP#1 closing lunch Sunday 2021-08-22, 1200
GLAMP#1 ends officially on to Sunday 2021-08-22, 1400

Location: Hacking Villa

FAQ

Where do I get Internet?

It is available everywhere at/around the Hacking Villa via WiFi. For cable based Internet bring a 15m+ Ethernet cable.

Where do I get Electricity?

You'll get electricity directly to the tent. Additionally the shared area also has electricity. You can also bring solar panels, if you like.

Where do I get food?

Breakfast is provided by us. But what about the rest of the day? There are a lot of delivery services available, ranging from Pizza, Tibetan, Thai, Swiss (yes!), etc. available.

Nearby are 2 Volg supermarkets, next Coop is in Schwanden, bigger Migros in Glarus and very big Coop can be found in Netstal. The Volg is reachable by foot, all others are reachable by train or bike.

There is also a kitchen inside the Hacking Villa for cooking. There is also a great barbecue place just next to the waterfall.

What can I do at the GLAMP?

There are alot of opportunities at the GLAMP:

You can ...

just relax and hangout
hack on project that you post poned for long
hike up mountains (up to 3612m! Lower is also possible)
meet other hackers
explore the biggest water power plant in Europe (Linth Limmern)
and much much more!

Automatic A and AAAA DNS entries with NAT64 for kubernetes?

2021-06-24T00:00:00Z

The DNS kubernetes quiz

Today our blog entry does not (yet) show a solution, but more a tricky quiz on creating DNS entries. The problem to solve is the following:

How to make every IPv6 only service in kubernetes also IPv4 reachable?

Let's see who can solve it first or the prettiest. Below are some thoughts on how to approach this problem.

The situation

Assume your kubernetes cluster is IPv6 only and all services have proper AAAA DNS entries. This allows you to directly receive traffic from the Internet to your kubernetes services.

Now to make that service also IPv4 reachable, we can deploy NAT64 service that maps an IPv4 address outside the cluster to an IPv6 service address inside the cluster:

A.B.C.D --> 2001:db8::1

So all traffic to that IPv4 address is converted to IPv6 by the external NAT64 translator.

The proxy service

Let's say the service running on 2001:db8::1 is named "ipv4-proxy" and thus reachable at ipv4-proxy.default.svc.example.com.

What we want to achieve is to expose every possible service inside the cluster also via IPv4. For this purpose we have created an haproxy container that access *.svc.example.com and forwards it via IPv6.

So the actual flow would look like:

IPv4 client --[ipv4]--> NAT64 -[ipv6]-> proxy service
                                         |
                                         |
                                         v
IPv6 client ---------------------> kubernetes service

The DNS dilemma

It would be very tempting to create a wildcard DNS entry or to configure/patch CoreDNS to also include an A entry for every service that is:

*.svc IN A A.B.C.D

So essentially all services resolve to the IPv4 address A.B.C.D. That however would also influence the kubernetes cluster, as pods potentially resolve A entries (not only AAAA) as well.

As the containers / pods do not have any IPv4 address (nor IPv4 routing), access to IPv4 is not possible. There are various outcomes of this situation:

The software in the container does happy eyeballs and tries both A/AAAA and uses the working IPv6 connection.
The software in the container misbehaves and takes the first record and uses IPv4 (nodejs is known to have or had a broken resolver that did exactly that).

So adding that wildcard might not be the smartest option. And additionally it is unclear whether coreDNS would support that.

Alternative automatic DNS entries

The .svc names in a kubernetes cluster are special in the sense that they are used for connecting internally. What if coreDNS (or any other DNS) server would instead of using .svc, use a second subdomain like abc.namespace.v4andv6.example.com and generate the same AAAA record as for the service and a static A record like describe above?

That could solve the problem. But again, does coreDNS support that?

Automated DNS entries in other zones

Instead of fully automated creating the entries as above, another option would be to specify DNS entries via annotations in a totally different zone, if coreDNS was supporting this. So let's say we also have control over example.org and we could instruct coreDNS to create the following entries automatically with an annotation:

abc.something.example.org AAAA <same as the service IP>
abc.something.example.org A    <a static IPv4 address A.B.C.D>

In theory this might be solved via some scripting, maybe via a DNS server like powerDNS?

Alternative solution with BIND

The bind DNS server, which is not usually deployed in a kubernetes cluster, supports views. Views enable different replies to the same query depending on the source IP address. Thus in theory something like that could be done, assuming a secondary zone example.org:

If the request comes from the kubernetes cluster, return a CNAME back to example.com.
If the request comes from outside the kubernetes cluster, return an A entry with the static IP
Unsolved: how to match on the AAAA entries (because we don't CNAME with the added A entry)

Support for IPv6 link local addresses in browsers

2021-06-14T00:00:00Z

Introduction

Link Local addresses (fe80::/10) are used for addressing devices in your local subnet. They can be automatically generated and using the IPv6 multicast address ff02::1, all hosts on the local subnet can easily be located.

However browsers like Chrome or Firefox do not support entering link local addresses inside a URL, which prevents accessing devices locally with a browser, for instance for configuring them.

Link local addresses need zone identifiers to specify which network device to use as an outgoing interface. This is because you have link local addresses on every interface and your network stack does not know on its own, which interface to use. So typically a link local address is something on the line of fe80::fae4:e3ff:fee2:37a4%eth0, where eth0 is the zone identifier.

Them problem is becoming more emphasised, as the world is moving more and more towards IPv6 only networks.

You might not even know the address of your network equipment anymore, but you can easily locate iit using the ff02::1 multicast address. So we need support in browsers, to allow network configurations.

Status of implementation

The main purpose of this document is to track the status of the link-local address support in the different browsers and related standards. The current status is:

Firefox says whatwg did not define it
Whatwg says zone id is intentionally omitted and and reference w3.org
w3.org has a longer reasoning, but it basically boils down to "Firefox and chrome don't do it and it's complicated and nobody needs it"
Chromium says it seems not to be worth the effort

Given that chain of events, if either Firefox, Chrome, W3.org or Whatwg where to add support for it, it seems likely that the others would be following.

IPv6 link local address support in Firefox

The progress of IPv6 link local addresses for Firefox is tracked on the mozilla bugzilla. The current situation is that Firefox references to the lack of standardisation by whatwg as a reason for not implementing it. Quoting Valentin Gosu from the Mozilla team:

The main reason the zone identifier is not supported in Firefox is
that parsing URLs is hard.  You'd think we can just pass whatever
string to the system API and it will work or fail depending on whether
it's valid or not, but that's not the case. In bug 1199430 for example
it was apparent that we need to make sure that the hostname string is
really valid before passing it to the OS.

I have no reason to oppose zone identifiers in URLs as long as the URL
spec defines how to parse them.  As such, I encourage you to engage
with the standard at https://github.com/whatwg/url/issues/392 instead
of here.

Thank you!

IPv6 link local address support in whatwg

The situation at whatwg is that there is a closed bug report on github and in the spec it says that

Support for <zone_id> is intentionally omitted.

That paragraph links to a bug registered at w3.org (see next chapter).

IPv6 link local address support at w3.org

At w3.org there is a bug titled Support IPv6 link-local addresses? that is set to status RESOLVED WONTFIX. It is closed basically based on the following statement from Ryan Sleevi:

Yes, we're especially not keen to support these in Chrome and have
repeatedly decided not to. The platform-specific nature of <zone_id>
makes it difficult to impossible to validate the well-formedness of
the URL (see https://tools.ietf.org/html/rfc4007#section-11.2 , as
referenced in 6874, to fully appreciate this special hell). Even if we
could reliably parse these (from a URL spec standpoint), it then has
to be handed 'somewhere', and that opens a new can of worms.

Even 6874 notes how unlikely it is to encounter these in practice -
  "Thus, URIs including a
   ZoneID are unlikely to be encountered in HTML documents.  However, if
   they do (for example, in a diagnostic script coded in HTML), it would
   be appropriate to treat them exactly as above."

Note that a 'dumb' parser may not be sufficient, as the Security Considerations of 6874 note:
  "To limit this risk, implementations MUST NOT allow use of this format
   except for well-defined usages, such as sending to link-local
   addresses under prefix fe80::/10.  At the time of writing, this is
   the only well-defined usage known."

And also
  "An HTTP client, proxy, or other intermediary MUST remove any ZoneID
   attached to an outgoing URI, as it has only local significance at the
   sending host."

This requires a transformative rewrite of any URLs going out the
wire. That's pretty substantial. Anne, do you recall the bug talking
about IP canonicalization (e.g. http://127.0.0.1 vs
http://[::127.0.0.1] vs http://012345 and friends?) This is
conceptually a similar issue - except it's explicitly required in the
context of <zone_id> that the <zone_id> not be emitted.

There's also the issue that zone_id precludes/requires the use of APIs
that user agents would otherwise prefer to avoid, in order to
'properly' handle the zone_id interpretation. For example, Chromium on
some platforms uses a built in DNS resolver, and so our address lookup
functions would need to define and support <zone_id>'s and map them to
system concepts. In doing so, you could end up with weird situations
where a URL works in Firefox but not Chrome, even though both
'hypothetically' supported <zone_id>'s, because FF may use an OS
routine and Chrome may use a built-in routine and they diverge.

Overall, our internal consensus is that <zone_id>'s are bonkers on
many grounds - the technical ambiguity (and RFC 6874 doesn't really
resolve the ambiguity as much as it fully owns it and just says
#YOLOSWAG) - and supporting them would add a lot of complexity for
what is explicitly and admittedly a limited value use case.

This bug references the Mozilla Firefox bug above and RFC3986 (replaced by RFC 6874).

IPv6 link local address support in Chrome / Chromium

On the chrome side there is a huge bug report which again references a huge number of other bugs that try to request IPv6 link local support, too.

The bug was closed by cbentzel@chromium.org stating:

There are a large number of special cases which are required on core
networking/navigation/etc. and it does not seem like it is worth the
up-front and ongoing maintenance costs given that this is a very
niche - albeit legitimate - need.

The bug at chromium has been made un-editable so it is basically frozen, besides people have added suggestions to the ticket on how to solve it.

Work Arounds

IPv6 link local connect hack

Peter has documented on the IPv6 link local connect hack to make firefox use fe90:0:[scope id]:[IP address] to reach fe80::[IP address]%[scope id]. Checkout his website for details!

IPv6 hack using ip6tables

Also from Peter is the hint that you can also use newer iptable versions to achieve a similar mapping:

"On modern Linux kernels you can also run

ip6tables -t nat -A OUTPUT -d fef0::/64 -j NETMAP --to fe80::/64

if you have exactly one outbound interface, so that fef0::1 translates to fe80::1"

Thanks again for the pointer!

The prettysocks SOCKS5 proxy (update 2021-11-24)

On 2021-11-23 we have been notified that there is a new workaround available: prettysock is a Socks5 proxy written in python that allows the use of Microsoft's ipv6-literal.net domain, but from any OS or browser, which is pointed to the proxy. So to access fe80::1ff:fe23:4567:890a%3, configure your browser to use the local prettysocks Socks5 proxy, replace the link local address with fe80--1ff-fe23-4567-890as3.ipv6-literal.net and there you go.

There are two interesting things to say about this solution:

It is a very simple solution
It is surprising that browser vendors haven't implement such a simple solution themselves so far - does it need an RFC that defines the domain to be used?

Other resources

If you are aware of other resources regarding IPv6 link local support in browsers, please join the IPv6.chat and let us know about it.

Making kubernetes kube-dns publicly reachable

2021-06-13T00:00:00Z

Introduction

If you have seen our article about running kubernetes Ingress-less, you are aware that we are pushing IPv6 only kubernetes clusters at ungleich.

Today, we are looking at making the "internal" kube-dns service world reachable using IPv6 and global DNS servers.

The kubernetes DNS service

If you have a look at your typical k8s cluster, you will notice that you usually have two coredns pods running:

% kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-gz5c7   1/1     Running   0          6d
coredns-558bd4d5db-hrzhz   1/1     Running   0          6d

These pods are usually served by the kube-dns service:

% kubectl -n kube-system get svc -l k8s-app=kube-dns
NAME       TYPE        CLUSTER-IP           EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   2a0a:e5c0:13:e2::a   <none>        53/UDP,53/TCP,9153/TCP   6d1h

As you can see, the kube-dns service is running on a publicly reachable IPv6 address.

IPv6 only DNS

IPv6 only DNS servers have one drawback: they cannot be reached via DNS recursions, if the resolver is IPv4 only.

At ungleich we run a variety of services to make IPv6 only services usable in the real world. In case of DNS, we are using DNS forwarders. They are acting similar to HTTP proxies, but for DNS.

So in our main DNS servers, dns1.ungleich.ch, dns2.ungleich.ch and dns3.ungleich.ch we have added the following configuration:

zone "k8s.place7.ungleich.ch"  {
   type forward;
   forward only;
   forwarders { 2a0a:e5c0:13:e2::a; };
};

This tells the DNS servers to forward DNS queries that come in for k8s.place7.ungleich.ch to 2a0a:e5c0:13:e2::a.

Additionally we have added DNS delegation in the place7.ungleich.ch zone:

k8s NS dns1.ungleich.ch.
k8s NS dns2.ungleich.ch.
k8s NS dns3.ungleich.ch.

Using the kubernetes DNS service in the wild

With this configuration, we can now access IPv6 only kubernetes services directly from the Internet. Let's first discover the kube-dns service itself:

% dig kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa

; <<>> DiG 9.16.16 <<>> kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23274
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f61925944f5218c9ac21e43960c64f254792e60f2b10f3f5 (good)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.k8s.place7.ungleich.ch. IN AAAA

;; ANSWER SECTION:
kube-dns.kube-system.svc.k8s.place7.ungleich.ch. 27 IN AAAA 2a0a:e5c0:13:e2::a

;; AUTHORITY SECTION:
k8s.place7.ungleich.ch. 13  IN  NS  kube-dns.kube-system.svc.k8s.place7.ungleich.ch.

As you can see, the kube-dns service in the kube-system namespace resolves to 2a0a:e5c0:13:e2::a, which is exactly what we have configured.

At the moment, there is also an etherpad test service named "ungleich-etherpad" running:

% kubectl get svc -l app=ungleichetherpad
NAME                TYPE        CLUSTER-IP              EXTERNAL-IP   PORT(S)    AGE
ungleich-etherpad   ClusterIP   2a0a:e5c0:13:e2::b7db   <none>        9001/TCP   3d19h

Let's first verify that it resolves:

% dig +short ungleich-etherpad.default.svc.k8s.place7.ungleich.ch aaaa
2a0a:e5c0:13:e2::b7db

And if that works, well, then we should also be able to access the service itself!

% curl -I http://ungleich-etherpad.default.svc.k8s.place7.ungleich.ch:9001/
HTTP/1.1 200 OK
X-Powered-By: Express
X-UA-Compatible: IE=Edge,chrome=1
Referrer-Policy: same-origin
Content-Type: text/html; charset=utf-8
Content-Length: 6039
ETag: W/"1797-Dq3+mr7XP0PQshikMNRpm5RSkGA"
Set-Cookie: express_sid=s%3AZGKdDe3FN1v5UPcS-7rsZW7CeloPrQ7p.VaL1V0M4780TBm8bT9hPVQMWPX5Lcte%2BzotO9Lsejlk; Path=/; HttpOnly; SameSite=Lax
Date: Sun, 13 Jun 2021 18:36:23 GMT
Connection: keep-alive
Keep-Alive: timeout=5

(attention, this is a test service and might not be running when you read this article at a later time)

IPv6 vs. IPv4

Could we have achived the same with IPv4? The answere here is "maybe": If the kubernetes service is reachable from globally reachable nameservers via IPv4, then the answer is yes. This could be done via public IPv4 addresses in the kubernetes cluster, via tunnels, VPNs, etc.

However, generally speaking, the DNS service of a kubernetes cluster running on RFC1918 IP addresses, is probably not reachable from globally reachable DNS servers by default.

For IPv6 the case is a bit different: we are using globally reachable IPv6 addresses in our k8s clusters, so they can potentially be reachable without the need of any tunnel or whatsoever. Firewalling and network policies can obviously prevent access, but if the IP addresses are properly routed, they will be accessible from the public Internet.

And this makes things much easier for DNS servers, which are also having IPv6 connectivity.

The following pictures shows the practical difference between the two approaches:

Does this make sense?

That clearly depends on your use-case. If you want your service DNS records to be publicly accessible, then the clear answer is yes.

If your cluster services are intended to be internal only (see previous blog post, then exposing the DNS service to the world might not be the best option.

Note on security

CoreDNS inside kubernetes is by default configured to allow resolving for any client that can reach it. Thus if you make your kube-dns service world reachable, you also turn it into an open resolver.

The following coredns configuration does correctly block requests, IF your coredns version is new enough:

  Corefile: |
    .:53 {
        acl k8s.place7.ungleich.ch {
             allow net ::/0
        }
        acl . {
              allow net 2a0a:e5c0:13::/48
              block
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
...

We tested this with coredns-1.8.4 in which the ACL behaviour is fixed.

More of this

We are discussing kubernetes and IPv6 related topics in the #hacking:ungleich.ch Matrix channel (you can signup here if you don't have an account) and will post more about our k8s journey in this blog. Stay tuned!

Building Ingress-less Kubernetes Clusters

2021-06-09T00:00:00Z

Introduction

On our journey to build and define IPv6 only kubernetes clusters we came accross some principles that seem awkward in the IPv6 only world. Let us today have a look at the LoadBalancer and Ingress concepts.

Ingress

Let's have a look at the Ingress definition definiton from the kubernetes website:

Ingress exposes HTTP and HTTPS routes from outside the cluster to
services within the cluster. Traffic routing is controlled by rules
defined on the Ingress resource.

So the ingress basically routes from outside to inside. But, in the IPv6 world, services are already publicly reachable. It just depends on your network policy.

Update 2021-06-13: Ingress vs. Service

As some people pointed out (thanks a lot!), a public service is not the same as an Ingress. Ingress has also the possibility to route based on layer 7 information like the path, domain name, etc.

However, if all of the traffic from an Ingress points to a single IPv6 HTTP/HTTPS Service, effectively the IPv6 service will do the same, with one hop less.

Services

Let's have a look at how services in IPv6 only clusters look like:

% kubectl get svc
NAME                                      TYPE        CLUSTER-IP              EXTERNAL-IP   PORT(S)                      AGE
etherpad                                  ClusterIP   2a0a:e5c0:13:e2::a94b   <none>        9001/TCP                     19h
nginx-service                             ClusterIP   2a0a:e5c0:13:e2::3607   <none>        80/TCP                       43h
postgres                                  ClusterIP   2a0a:e5c0:13:e2::c9e0   <none>        5432/TCP                     19h
...

All these services are world reachable, depending on your network policy.

ServiceTypes

While we are at looking at the k8s primitives, let's have a closer look at the Service, specifically at 3 of the ServiceTypes supported by k8s, including it's definition:

ClusterIP

The k8s website says

Exposes the Service on a cluster-internal IP. Choosing this value
makes the Service only reachable from within the cluster. This is the
default ServiceType.

So in the context of IPv6, this sounds wrong. There is nothing that makes an global IPv6 address be "internal", besides possible network policies. The concept is probably coming from the strict difference of RFC1918 space usually used in k8s clusters and not public IPv4.

This difference does not make a lot of sense in the IPv6 world though. Seeing services as public by default, makes much more sense. And simplifies your clusters a lot.

NodePort

Let's first have a look at the definition again:

Exposes the Service on each Node's IP at a static port (the
NodePort). A ClusterIP Service, to which the NodePort Service routes,
is automatically created. You'll be able to contact the NodePort
Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.

Conceptually this can be similarily utilised in the IPv6 only world like it does in the IPv4 world. However given that there are enough addresses available with IPv6, this might not be such an interesting ServiceType anymore.

LoadBalancer

Before we have a look at this type, let's take some steps back first to ...

... Load Balancing

There are a variety of possibilities to do load balancing. From simple round robin, to ECMP based load balancing, to application aware, potentially weighted load balancing.

So for load balancing, there is usually more than one solution and there is likely not one size fits all.

So with this said, let.s have a look at the ServiceType LoadBalancer definition:

Exposes the Service externally using a cloud provider's load
balancer. NodePort and ClusterIP Services, to which the external load
balancer routes, are automatically created.

So whatever the cloud provider offers, can be used, and that is a good thing. However, let's have a look at how you get load balancing for free in IPv6 only clusters:

Load Balancing in IPv6 only clusters

So what is the most easy way of reliable load balancing in network? ECMP (equal cost multi path) comes to the mind right away. Given that kubernetes nodes can BGP peer with the network (upstream or the switches), this basically gives load balancing to the world for free:

                            [ The Internet ]
                                   |
     [ k8s-node-1 ]-----------[ network ]-----------[ k8s-node-n]
                              [  ECMP   ]
                                   |
                             [ k8s-node-2]

In the real world on a bird based BGP upstream router this looks as follows:

[18:13:02] red.place7:~# birdc show route
BIRD 2.0.7 ready.
Table master6:
...
2a0a:e5c0:13:e2::/108 unicast [place7-server1 2021-06-07] * (100) [AS65534i]
    via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
                     unicast [place7-server4 2021-06-08] (100) [AS65534i]
    via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
                     unicast [place7-server2 2021-06-07] (100) [AS65534i]
    via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
                     unicast [place7-server3 2021-06-07] (100) [AS65534i]
    via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
...

Which results into the following kernel route:

2a0a:e5c0:13:e2::/108 proto bird metric 32
    nexthop via 2a0a:e5c0:13:0:224:81ff:fee0:db7a dev eth0 weight 1
    nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 dev eth0 weight 1
    nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 dev eth0 weight 1
    nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc dev eth0 weight 1 pref medium

TL;DR

We know, a TL;DR at the end is not the right thing to do, but hey, we are at ungleich, aren't we?

In a nutshell, with IPv6 the concept of Ingress, Service and the LoadBalancer ServiceType types need to be revised, as IPv6 allows direct access without having to jump through hoops.

If you are interesting in continuing the discussion, we are there for you in the #hacking:ungleich.ch Matrix channel you can signup here if you don't have an account.

Or if you are interested in an IPv6 only kubernetes cluster, drop a mail to support-at-ungleich.ch.

IPv4 as a service (ungleich tech talk #4)

2021-03-25T00:00:00Z

In the fourth ungleich tech talk you can see how to get IPv4 via IPv6 using the IPv6VPN and NAT64.

We hope that you enjoy this video and if you have any questions, feel free to join the IPv6 chat. You can also read more about IPv4-as-a-service.

An Introduction to NAT64

2021-03-09T00:00:00Z

In the third ungleich tech talk we give an introduction on NAT64 and how it works in general.

We hope that you enjoy this video and if you have any questions, feel free to join the IPv6 chat.

Accessing IPv4 only hosts via IPv6

2021-02-28T00:00:00Z

In the second ungleich tech talk you can find out how to access IPv4 only devices from the IPv6 Internet using a simple NAT64 translator.

This is especially helpful if you are by default in IPv6 only networks, but you still have legacy-only networks or legacy-only devices that you need to access.

In this ungleich tech talk we use the VIWIB as a NAT64 translator and a reconfigured VIGIR as an IPv4 only demo device.

If you have any questions, feel free to join the IPv6 chat.

How to store your data without CO2 emission

2021-02-25T00:00:00Z

When we talk about climate change or sustainability, we usually think about things we can see such as factories, cars, or plastic waste. What we don't notice though is how much we actually contribute to climate change by our digital behaviour.

Even when we don't notice, we are still respopnsible for the consequences of our action. So what is the current state of digital data storage, and what can we do to change things for the better?

How data is stored

Your data is usually stored in one of the many data centers in the world. To save your data, multiple disks or even multiple servers are used to ensure that your data is safely stored.

That means When you open up your mobile phone to browse pictures or to read your email, you access the servers to retrieve your data.

How data centers operate

For servers to run they need electricity, on one hand for actually running the servers that store your data and on cooling on the other hand. The amount of energy used for cooling is quite significant, too. And this is an imoprtant question for the environment: how is the electricity for running and cooling the servers produced, with what source?

Unfortunately, most of datacenters don't run only on renewable energy. Most of them use energy from the grid, and this energy can be produced by coal, nuclear power or renewable energy.

On better cases datacenters use a mix of energy sources, making their energy source portfolio somewhat green. Some datacenters do buy CO2 certificates to compensate for this fact.

So this is the current state, even if you are living a very green and sustainable way by saying no to plastic, meat, and recycle as much as you can, you can still be contributing to CO2 emissions by using your default data storage. Because the default energy source of digital world is not green, at better cases it is somewhat green by mixed source or a bit green by offsetting the CO2 emission.

An alternative: the sustainable way of saving data

So how can we tackle this situation? Our project of Data Center Light came into the world by questioning the status quo.

And this is how it is done:

instead of buying energy, we produce it locally (using PV and a local hydropower plant)
instead of offsetting by buying certificates, we use 100% renewable energy
instead of building new data centers, we reuse existing factory halls
instead of buying new servers, we buy 2nd hand (with new disks/ssds)

The sustainable alternative

This way we want to set the new default that is as little energy as popssible, and as green as energy as possible. By redesigning how our servers operate there is practically no CO2 emission from power consumption and also the grey energy is minimised.

With this setting, we are creating a place where everybody can make a sustainable choice in storing their data. Where one can save the data in a green storage at our Zero Carbon Cloud, or run Virtual Private Servers at our Data Center Light, back up their data with green energy, talk to each other via sustainable chat such as Zero Carbon Matrix or Zero Carbon Mattermost.

Since the beginning of our journey we have received a lot of love and support from the tech and sustainability community. Our team at ungleich is very proud of our growing range of products, because it was not only the result of our own effort that added a new product on the list. It was very much reflecting the need of the community that supported us, that demanded greener, more sustainable choices to be available.

Call to action

We are a team of people who believe that there are really a lot of things that can be done better. You can make big changes by simply start caring for things that you did not before. That's why we urge everybody to think about the digital data we're using daily - how and which energy is it using, and is it sustainable.

If it is based on sustainable sources, support it, give it love so it can continue. If it is not based on sustainable sources, demand it to do better. We already have solutions, we just need everybody to do their part.

If you have any thoughts to share, you can reach us at sustainability at ungleich.ch or in the ungleich chat.

The v6 pattern for IPv6 only hosts

2021-02-24T00:00:00Z

At ungleich we have a lot of IPv6-only web servers. Many of them are are proxied from the IPv4 world, so the domain name points to two different machines:

the AAAA entry points to the server directly
the A entry points to a proxy

This sometimes makes configuring the right system a bit harder, because on dual stack clients, accessing www.example.com brings you to either machine. In the first ungleich tech talk we show how this looks in detail and how we ensure that we configure the right machine.

This is our first tech talk and we love to hear your feedback.

A love letter to ISC bind

2021-02-22T00:00:00Z

Dear ISC bind,

this is a love letter to you. You probably don't know me, but I have been a long term user of yours.

I started my time with you in the late 90's. It was when you were called "bind 4". I was very happy with our relationship. You'd not only take care of all authoritative requests, but also take care of caching client requests. Me, still being young at the time, I did not know nor care about security concerns in the beginning.

But then over time I got more experienced and I read and tried DNS cache poisoning and I was shocked. How could you? How could you accept incorrect entries? I had so much trust in you and then that!

Years passed and after my shock, I had a fling with djbdns (together with qmail and daemontools). Which right away took security more serious. So serious that even managing djbdns with its own suite was almost like a crypto analysis adventure (no offense, Dan!). Many years this was my software solution of choice, compiled by source, patched by hand. Oh, the old 2000's!

Over time the effort for managing software by source code and /usr/local installations did not turn out to be very efficient. So I looked around and found powerdns, nsd and unbound.

I settled for the nsd/unbound combination for many years. Solid, easy to use and nice separation of concerns. Thanks nlnetlabs! Then I stumbled upon dnsmasq. Dnsmasq feels a bit like a younger sibling of bind: it does everything and even includes dhcp and tftp support! Crazy, isn't it? Many years to come, dnsmasq, first discovered on an embedded router, turned out to be a very stable solution for even mid sized installations. And it comes with a very simple configuration as well.

But then 2017 happened. And ungleich started the Data Center Light project. An IPv6 first hosting. And there you were, dear bind. Looking at me from the side of the software projects, saying "I think it's time we have a talk.".

And indeed, we did have a talk. A talk about implementing DNS64. About different DNS64 prefixes in one configuration. About being an authoritative name server that functions even if all upstreams are down. A name server that even allows the most funky configuration of removing native AAAA entries for DNS64 networks that should only access mapped IPv4 addresses. You can do it all, but you are still not complicated. Who can say that from oneself?

I admit, I was not always loyal to you. And I also admit that I am still sceptical about mixing caching and authoritative features in one process. But you do it so damn well. Not only have you been around for decades and collected the wisdom over the years, but also have you adapted to the time.

This is why I am writing you this love letter today, to say thanks. Thanks for making the life in a data center easier, thanks to being flexible, thanks for improving over time and thanks to still adhearing to the same configuration file format that I used in the late 90's.

Dear BIND, you are by far not perfect, but then neither is reality. And this is your strength, solving real world problems.

Thank you for doing so and thanks to all the involved developers for creating bind.

In love, yours,

Nico

The chat app of 2021

2021-02-09T00:00:00Z

Introduction

At this point we've all seen many chat apps - some very popular and some not. Some very secure and some not. Some very centralised and some not. The difficult thing when choosing a chat app is that the most important differences are not something you can intuitively find out. That is more a question of how each chat app was designed from the beginning.

What we are going to do in this post is that we will talk about what we have used as chat so far and why, and we will discuss what you need to consider when making a decision for yourself. What are the differences between the chat apps out there, and what do those differences mean to their end-users?

About us

Each community or individual has a different need for the chat and what's best for others does not necessarily mean that's going to be the best for you. Before starting we want to give you an idea about who we are.

Our team is rather deeply into technology probably more than most people.
The age of our chat users is somewhere between 18-55 (with minor exceptions).
Many of our chat users are working with laptops for a big part of the day.
Most of our chat users are comfortable using English for communication.
Our chat users are about 80 % male and 20 % female.
A big part of our community values sustainability and the environment
A big part of us is interested in software with its code publicly open - in other words, open source.

The evolution of our chat was really finding what fits us the best. We have started our team chat at Slack many years ago, then we moved to Rocketchat, and then we moved to Mattermost. While keeping Mattermost running, we started our Matrix instance on the side. We have gradually moved more to Matrix over time and now have most of our main conversations in Matrix. There were reasons for each of our moves, we will explain them below.

What we based our decisions on

Work chat

We (ungleich) are a company working with Free and Open Source and Linux. Our work happens around many text information and we have many parallel projects shared with people from different timezones. This requires our chat to be as efficient and organised as possible, so that everybody can get their work done and collaborate easily. This was the reason why we were initially landing in Slack which has an interface suitable for our work life. Apps designed for more casual, mobile use such as Telegram and Signal did not fit our need.

Self-hosted

We need to mention that for a Swiss company like ourselves that physically runs our servers with local hydropower, we knew that it is senseless to have our chat somewhere in the U.S. (whose privacy laws are more relaxed than ours) run by servers with fossil fuels.

That's why we soon moved away from Slack to chats that we can host on our own. We first moved to Rocketchat, and although the transition was easy and smooth at the beginning its mobile version was not as smooth as we liked it to be at that time and our team would miss getting messages often while on the run between different data center locations.

After spending some time with Rocketchat we moved to self-hosted Mattermost looking for a more stable experience. With its robust performance and friendly UI (that we still think is one of the best out there), our team became quite happy with Mattermost.

Decentralised

You might have noticed that we are not the most typical company: renewable-only energy driven Linux, FOSS and IPv6 aficionados in the Alps. This means we attract quite some people who have a lot of stories and chats to share with us. More and more over the time it became clear that our work chat is not only for our internal work - but it's also for a community chat, a community with very diverse backgrounds for that matter. Hobby computer lovers, mountain lovers, hackers and makers, climate activists, penguin lovers, you name it.

While happily staying on Mattermost we started to receive requests from our community for enabling our chat to be more decentralised: meaning not all the chat data staying in our server only (centralisation), but enabling others to connect from their servers too (decentralisation).

This was a fair request and we knew it was the right way to go. As a solution we started to phase in Matrix into our chat - we first started it as a side chat from our main channels in Mattermost.

Over time we moved most of our conversations to it, while keeping Mattermost on the side. Using Matterbridge to bridge between the two chats, this dual system has been working quite well, and it is especially useful when one of the instances experiences downtime.

What you should consider when choosing a team chat

Above was our specific case and you probably have a different use for the chat app. Should you consider Matrix? Mattermost? Or Slack? Some questions need to be answered first to see what your requirements are.

1. Workspace or SMS

This depends on what your use case is. Are you looking for a secure version of SMS, or are you looking for an organised place for different groups and tasks to be handled? The former fits the use case of Threema, Signal, Telegram, Whatsapp, and the latter is better handled by Slack, Mattermost, Rocketchat, and Matrix.

2. Can you be anonymous

This is closely linked to the #1, because the chat apps that function as SMS replacement such as Signal, Telegram, Whatsapp, need a phone number to work. So if staying anonymous is an important factor for you this is something you want to consider in choosing a chat. For using Matrix or Mattermost you do not need a phone number, and in the case of Matrix you do not even need an email address that can identify you.

3. Where is the chat server

This is something most people don't think about: the physical location of the servers where the chat is running. It mostly becomes a crucial question in two ways.

One is if you face the possibility of having to deal with authorities - then you need to know which law will apply in case you are being investigated.

Two is if you care for the environmental and ethical impact of your servers, such as what kind of energy source is being used and what is the ethical stance of the hosters.

The ability to decide where your servers will physically be, either by running the server by yourself or by somebody you can trust is a big difference between some chat apps and others. Chats that offer home servers, self-hosted options such as Matrix or Mattermost give you the ability to choose, and any chat app that does not, locks you in the choice of the chat provider.

4. Decentralised or centralised

Does all of our conversation only go to one company or to one country, or can it be in the different small or big streams everywhere and be independently open to each other? Most chats out there are the former, and federated chats are the latter.

Say there's a special house where everybody goes to for communication, and you have to go there whenever you want to talk to anybody, and the conversation stays only in that house. Anybody who wants to say anything to anybody, has to go into this house. Doesn't it sound really strange? That's what most chats are.

But what if we can stay in our own home and call each other and still can have a conversation? And the conversation does not only stay in one house, but in everybody's house when it's their own conversation? That's more like how Matrix imagined how chats should work.

This is the single most standout identity of Matrix, that it has built-in federation. Means you can stay on your chat and go to other Matrix instance and talk to them.

5. Are you into the latest, hottest technology

It might sound silly but it is still an important factor for a lot of early adopters and people who just gotta have their hands on the latest groundbreaking stuff (cough). This is it. Matrix is the hottest chat system of 2021 and hands down the most innovative project amongst all the chat apps out there because of its federation and security design.

The baselines that you shouldn't compromise

A lot of us brush off the topic of privacy thinking "But I don't have anything to hide." but it is not that simple. Even though you might not have anything to hide now, people who you have a conversation with might have a very different stance and you affect them with your choices.

Using a chat app for communication involves the privacy of everybody in your network - family, friends, colleagues, and more. That's why whatever chat you choose, there should be some baselines you shouldn't compromise. These are what we think is important in choosing which chat to use in 2021.

The baseline 1: End-to-End Encrypted

As a very base your chat should be End-to-End Encrypted (E2EE). E2EE means even when the 3rd party (including law enforcement or the hosting company itself) snoops into the chat data, they will just see a series of useless encrypted texts that can not be decrypted.

Not all chats have proper E2EE and especially not all chats have it as default. Matrix has E2EE as a default for example, and Telegram does not. It works quite the opposite way in fact: disabling E2EE has to be manually opted in Matrix, whereas enabling E2EE can be done only by choosing "Secret Chat" Function in Telegram.

The baseline 2: Not collecting your data

Even when the chat app can not read your conversation thanks to the encryption, some chats do access a lot of other information, such as with whom you are talking to for how often and for how long ("meta data"). We tend to focus on not revealing the content of our chat and to forget that the other information can be collected while being unnoticed.

Whatsapp is a particularly risky choice for this reason, because it belongs to Facebook whose entire business model relies on collecting user information for monetisation.

The baseline 3: Is the code open for public

The thing about closed code is that nobody outside the chat app company can see how and what is built in the chat. Is it having some back doors users are unaware of, is it monitoring the user activity without letting the user know, with closed code we will never know.

So for a chat (or any software for that matter) to be claimed secure, its code has to be open for the public, so unbiased third parties can review its sanity. If you think about it, it's quite simple - no system can achieve robust integrity without transparency.

Try it yourself

When it comes to technology, the best approach is trying it yourself. Our ungleich chat, both Matrix and Mattermost, are open for anybody to join: we heartily invite you to give it a try. Create an account, join rooms, say hi and ask questions you might have.

We claim our chat is one of the safest places to try a chat app you are not sure about yet - it does not collect your data, it does not need your phone number, it runs on 100% renewable energy.

More about us and Matrix

No IPv6 in 2021? Not acceptable.

2021-01-09T00:00:00Z

It's 2021 and it's time to stop using legacy IP (IPv4) only services. In 2020 we have already seen quite a rise in IPv6 connectivity, as well as more and more services emerging IPv6 only.

The IPv6 community is very active

The IPv6 community has reported many bugs to products like Steam from Valve, Dockerhub and so many more that it was not possible for me to list them all in this article.

As a matter of fact, websites or services that have degraded or non-existing IPv6 connectivity are reported daily on Twitter.

Time to move on

As I can see it from the work at ungleich, 2020 was a very active year when it comes to IPv6. But at some point, services that don't work with IPv6 are simply not worth caring about anymore.

While IPv6 has been defined in 1998, it is clear that vendors and providers need some time to adapt. But 23 years should have been more than enough to migrate.

For this reason we want to encourage everyone in 2021 to move on, to cut off the old legacy IP applications, services or hardware.

How to move on?

There are a variety of things everyone of us can do and we have collected some easy to take steps.

Setup networks IPv6 only

For every new network you setup, set it up with IPv6 only. All current mobile phones, tablets and operating systems work in IPv6 only networks.

Setup services and applications IPv6 only

If you are an application developer, I suggest to develop your application with IPv6 first. And if you setup a new service, configure IPv6 first and maybe only.

It's important to understand that setting up something IPv6 only, does not mean it will be unreachable from the legacy IP Internet. NAT64 is one easy to use technology to solve this problem.

Abandon legacy only

Whether it's network hardware, applications, Internet providers, hosting providers - stop using them if they don't work in IPv6 only environments. And let the everyone know that you did so.

While I am not a fan of shaming anyone, it is still important to let vendors know that you have moved on because of the lack of IPv6. Otherwise the vendor does not know why they are losing customers. When you communicate that you abandon a product or service because of lacking proper IPv6 support, I recommend to doing so in a clear, but respectful way.

Advocate IPv6 only

To be able to spot bugs and to make life for network operators easier, I strongly recommend going IPv6 only. Not dualstack, but simply IPv6 only. If you deploy NAT64 in IPv6 only networks, you can even create reachability in both directions.

Many of us network operators, developers and sysadmins are very good in implementing things. However, it is important to share the word.

Help each other

For about 2 years there is an open, community driven IPv6 Chat. In this chat you can ask others for help, ask which alternative products or services to use. Or how to migrate to IPv6 only networks. There is also an IPv6 reddit and an IPv6 group on Facebook. Many of the RIRs (RIPE, APNIC, Afrinic, LACNIC and ARIN) also offer IPv6 training resources.

Migrating to IPv6 is not difficult and there is a community to help you with it.

Let's do it, together.

Managing IPv4 islands with Jool and OpenWrt

2020-12-15T00:00:00Z

Introduction

At ungleich we are using Jool in a variety of scenarios with NAT64 or SIIT. The main use of jool in our infrastructure is to enable IPv6 only hosts to communicate with the IPv4 Internet.

However today we want to show you a different use case of jool: Enabling IPv4 islands to communicate with the IPv6 Internet.

For this we will focus on using Jool on OpenWrt, because this is a platform that you can also easily use in your networks or even at home.

The general problem to solve

The literally biggest problem to solve when connecting the two different worlds is that the IPv6 space is significantly bigger. This is a problem, because we cannot achieve a 1:1 mapping from the IPv4 world, but we can do a 1:1 mapping from the IPv6 world:

Installing jool

Installing jool on OpenWrt is very easy, it is just a matter of installing the kernel module and the tools for managing jool:

opkg update
opkg install kmod-jool jool-tools

Making IPv4 islands reachable

Assume that you are mostly running IPv6 only networks. And you happen to have some hosts, which, for whatever reason, cannot be switched to IPv6. We can use a stateful NAT64 to map "the whole IPv6 Internet" to 192.0.2.1 as follows:

This works pretty similar to regular NAT that you are used from home. If we compare it visually, it is even more clear:

Let's have a look at this in an OpenWrt context:

The LAN network is usually 192.168.1.0/24
The router's IPv4 address is usually 192.168.1.1
In this example we routed 2a0a:e5c1:18f::/48 to the router
192.168.1.0/24 has 8 bits for the hosts (32-24=8)
We choose 2a0a:e5c1:18f:b00::/120 to map the IPv4 island (128-120=8)
We use the OpenWrt's standard address to masquerade/squash the IPv6 Internet

First we will create an "IPv4 pool":

root@vigir2:~# jool -4 -a 192.168.1.1
root@vigir2:~# jool -4
+------------+-------+--------------------+-----------------+-------------+
|       Mark | Proto |     Max iterations |         Address |       Ports |
+------------+-------+--------------------+-----------------+-------------+
|          0 |   TCP |       1024 ( auto) |     192.168.1.1 |     1-65535 |
+------------+-------+--------------------+-----------------+-------------+
|          0 |   UDP |       1024 ( auto) |     192.168.1.1 |     1-65535 |
+------------+-------+--------------------+-----------------+-------------+
|          0 |  ICMP |       1024 ( auto) |     192.168.1.1 |     0-65535 |
+------------+-------+--------------------+-----------------+-------------+
  (Fetched 3 samples.)

This allows jool to map IPv6 addresses stateful to 192.168.1.1 and basically allows incoming IPv6 traffic. What is left now is to configure the mapping from IPv6 to IPv4. For this we use the pool6 argument of jool:

jool -6 2a0a:e5c1:18f:b00::/96

Note that we cheated here. We did not only map 2a0a:e5c1:18f:b00::/120, but we did actually map the whole IPv4 range. The advantage of this is that we do not need to care which networks are used on the IPv4 island. Any IPv4 address inside the LAN segment is now reachable. If you want to reach the IP address 192.168.1.42, you can ping as 2a0a:e5c1:18f:b00::192.168.1.42. As a matter of fact, while writing this article, the sample network is up and running and you should be able to ping 2a0a:e5c1:18f:b00::192.168.1.1 from the IPv6 Internet.

More of this?

If you are interested in IPv6 or network, feel free to join us on the IPv6.chat.

An alternative to annoying phone hotlines

2020-11-29T00:00:00Z

The phone hotline

If you have a problem with your contract, if you want to have some information about your product or you just want to change some detail. By default, many companies nowadays offer something that we call "hotline". Or in other words: a voice based communication that allows to easily queue people.

The motivation and how it works is rather clear: there are a finite number of employees, each of which can only talk to one person at a time. So if the number of requests is more than number of employees, then you are stuck in the queue. However this post is not about the annoyance of waiting in a queue and enduring little quality, down sampled classic music pieces.

No, in this post I want to address a different fundamental problem: every time you call, you start a fresh conversation. You are likely to talk to a different person with a different background and probably not much knowledge of your situation. From a customer perspective you usually don't have any trail of previous communication. Actually, the company you are calling might record (and correctly announce it before) the call. However as a customer, you cannot easily record as well. Often it is also impossible to correspond with the company by email, so all written communication has to be sent in letters. In 2020!

Looking at it this way clearly shows how much power imbalance the innovation of the phone hotline is causing. But it could easily be different.

The classic way

You might or might not remember when companies used to be smaller and you would request a service with the counterpart in person. Both of you know each other and are fully aware of each other responsibilities ("I give you money, you give me a product or service") and also of the support process (A: "It did not work" - B: "I'll fix it!). Often the service provider was not far away, might even have been my neighbour. And it's really not good for our relationship if my neighbour does do what he promised to do.

As you can easily imagine this does not scale nor work easily in big companies where staff is rotated or fluctuating. The old value system of being responsible on a personal basis cannot easily be transferred. This also means that the classic way is much more expensive in terms of time and resources, but the responsibilities are enforced by social relationships.

Mixing the two?

So we could say that they are two extremes: one very personal, high quality, expensive and the other - well, you get the picture. Is it possible to improve the current situation and how can we get the best of the two worlds? Before answering this question, let me give you a short background of where we, ungleich, are and how we work, to show you how these approaches can naturally merge.

ungleich @ Digital Glarus

ungleich is based in Digital Glarus, a really old mountain valley in Switzerland. Majority of its buildings are very old (I'd guess most are built prior to 1900, many even much older), major businesses are industry, farming and also tourism. Many people here get up before 6 and start working latest by 8.

We from ungleich on the other hand are working in IPv6 only networks connected by our own fiber or with long range wifi links. Our working hours are very flexible, can be morning, day, night, week, weekend - we are free to choose. Our topics are very technical by nature.

These two approaches can contradict, but they can also work together very well. Like the two ways of communication.

Interestingly our experience here is that they can easily be combined: many people living in Digital Glarus have what we call an "old value system". If you offer a service towards people in Digital Glarus, you need to take responsibility and be trustworthy. Otherwise the word will get out within a few days and social enforcement will result in no more work for you. While this might sound cruel, you could actually call this "social quality assurance". Actually a bit similar to what we see in social media, just lower scale.

And how does this look like in reality? People here want and need to be convinced that you are trustworthy. You are having in person meetings (before corona), one person will make a protocol and then later send it for verification back to the other party. If something is noted incorrectly, the protocol will be amended and again verified.

This ensures that trust is built and also that both parties, the delivering company as well as the customer are playing on eye level.

Combining old values and new communication

Let's come back to the original problem: we shifted from high quality, individual services to mass produced in-transparent communication. Technically and organisational, it is not necessary to provide a worse product or service if it is mass produced. It just happens to be the case due to technical limitations in the beginning.

So let's go back to the hotline problem: we advocate a simple change that costs little for companies to implement but restores trust and quality in communication:

Every support hotline should be, by default, accompanied by a text based ticketing system that sends users a protocol and let's them interact with you on a text basis.

So how does this work? The agent in the call center will make notes of the phone call - they are already done nowadays, but unavailable for you. Some of these notes might be internal ("The customer does not know the difference between the power button and the reset button - always advise to push the button on the right") and are not for sharing. However, the key points of the conversation must be sent to the customer. This way, as a customer I can easily react and correct statements that have been incorrectly recorded. With a trail.

Furthermore in a later stage, as a customer, I also have a trail and the ability to respond to the previous conversation by text, giving me the opportunity to add to the trail. And to built trust on the way.

Obviously, our suggestion here is not rocket science. In fact, it is a very easy, natural and cost effective measure to be more transparent and to built mutual trust.

Some companies might try to argue that it is too complex or too expensive to implement such a system. To prevent that argument from being true, we have added a Hosted Support System to our product list. Nobody needs to get it from us, but anybody can. And thus there is no excuse, not to have it implemented. It is a very similar approach to not have an excuse for not having IPv6, but that is a story for another day...

The new EU draft endangers everyone's security

2020-11-09T00:00:00Z

TL;DR

The EU is trying to disable encryption for everyone. However, this approach is fundamentally flawed, as the bad guys don't follow the law.

Introduction

The Council of the European Union has published a draft which requires everyone who is offering secure communication channels to allow authorities to read the communication.

The motivation is clear: terrorist attacks and unlawful behaviour should be prevented by wiretapping. No crime is better for everyone. So far, so good. In theory.

First problem: reducing security affects everybody

The first problem is that modern encryption is not easy to break, or let's put it clearly: it is almost impossible to break. Thus passing this law requires decades of work to be undone. To make systems that have been mathematically proven to be secure, more insecure.

This reduces security for any communication by default. And this does not only affect terrorists, but also government agencies and the general public.

Thus it also reduces the freedom of speech. There are activists out there (f.i. in the area of climate change) that fear their life, if their communication is revealed, because some governments do not allow free speech.

Second problem: the bad guys don't comply

One of the strangest problems with the EU proposal is that the idea is to make this into a law that everyone has to follow. Or, more precisely: the idea is that companies like Whatsapp or Signal have to provide keys or backdoors into their systems that authorities can use for wiretapping.

Now, this is a crucial problem. Because companies like us, ungleich, also provide secure communication using Matrix. And we are not in the EU (fact check: Switzerland is not in the EU).

See the problem? No? Well, let's say you are the bad guys and you plan to coordinate some attack. What do you do?

You run your own chat system. It is very easy to do. It cannot be technically prevented. It might be against the law in the EU to run a chat system that does not allow backdoor access, ok. But then again - you are going to do something that is against the law anyway. So this is the least of your problems.

So the proposed law is actually doing the opposite of its intention:

It reduces security for everyone who is behaving according to law
It does not prevent unlawful parties from communicating securely

Third problem: criminalizing science

Apart from the obvious two really strong problems, the law might actually lead to research and science being prohibited. The underlying algorithms are usually based on mathematically hard-to-solve problems.

The problems are carefully researched and in the end used to provide security, confidentiality and integrity.

Researchers can be hindered by legal questions whether or not they are able to solve mathematical problems. Which then again can and will stop the progress in other areas of science as well. This all sounds terribly wrong, doesn't it?

Fourth problem: a new attack vector

Let's assume for a moment that none of the above problems is already crucial enough to stop the whole motion. There is one more big and crucial problem: if authorities have a backdoor into your communication, this backdoor needs to be submitted to the authorities. It needs to be securely stored by authorities.

It means that this law will make authorities a very interesting target for hacking into. You do not need to attack a technically very secure system. You can just hack the authorities server and you gain access to everyone's communication.

This enables much easier access for terrorists, foreign (enemy) governments and everyone else who is interested in getting access to your communication.

Summary

The proposed draft is dangerous for everyone except the criminals. It is dangerous for civilians, governments, journalists, whistle-blowers and even the science and medical sectors.

The whole approach is fundamentally flawed and if passed as-is reduces security for everyone, but the bad guys.

We urge everyone reading this article to do whatever is in their power to stop this law passing, before it is too late. And too late might unfortunately already be on the 19th of November 2020.

Related websites

The IPv6-eye: a proper IPv6 camera

2020-11-04T00:00:00Z

TL;DR

At ungleich we have created an IPv6 camera that brings its own IPv6 connectivity. And a world wide reachable name. We are currently crowdfunding the production of the first 100 cameras.

A real IPv6 camera?

What is the advantage of IPv6 again? Exactly, world wide reachability without any hacks or quirks.

To support this, we have not only created a camera that works within all types of networks:

IPv6 only
IPv4 only
Dual Stack (IPv6+IPv4)

We took it one step further: The IPv6-eye is an IPv6 camera that brings its own IPv6 connectivity. This way it works independently of whether your network actually supports IPv6 or not.

On top of all of this, we created the 1000ey.es project that even gives a cool name to every IPv6 camera in the world.

How to use the IPv6-eye

It is very simple:

plugin a network cable with Internet uplink
connect it to power
It works!

After that you can ...

Point the camera to what you want to show
Visit yourname.1000ey.es:8081 (you can find an example on mountains-diesbach.1000ey.es
Optionally configure it to connect to your existing wifi network

How does the IPv6-eye work?

But what happens under the hood? The IPv6-eye is based on a similar concept as the VIIRB. The IPv6-eye uses the IPv6VPN to acquire IPv6 connectivity. This works in IPv4 only, IPv6 only and dual stack networks.

From that moment on, the camera is world wide reachable.

A name for every IPv6 camera out there

However, people do not like to type IP addresses. Not IPv4, not IPv6. For this reason you can choose a name below the domain 1000ey.es for your camera. This offer is for free on top of the actual hardware.

Fully Open Source

Like the VIIRB, the whole stack is fully Open Source. The operating system is OpenWRT. The VPN software is wireguard. And the video streaming server is based on mjpg-streamer or motion (still has to be decided for the final shipment). All files that are required to configure the IPv6-eye can be found in the ungleich-tools repository.

Crowdfunding the production

As it is usual for the production process to start at a minimum size, we aim to crowdfund the first 100 units. If we exceed the minimum goal of 100 IPv6 eyes, we have added some extra goodies as an option.

Checkout the crowdfunding campaign for more details.

Encrypted rootfs with Alpine Linux

2020-10-08T00:00:00Z

Introduction

This is a short guide on how to encrypt your root filesystem on Alpine Linux. This article assumes an EFI based system.

Booting Alpine Linux

Use the standard Alpine Linux installer to boot. Prepare networking and and apkrepos:

setup-interfaces

If you are in an IPv6 only network, setup a nameserver. At the moment Alpine Linux does not start rdnssd by default. The following works for VMs on Data Center Light

echo nameserver 2a0a:e5c0:2:a::a

Then setup the repos:

setup-apkrepos

Optional, if you want to continue the installation remotely from another computer via ssh:

setup-sshd

And then add your ssh key to /root/.ssh/authorized keys. We are using the key.wf service for staff at ungleich:

mkdir -p /root/.ssh/
wget -O ~/.ssh/authorized_keys  key.wf/nico

Create partitions

In this guide we assume you create 3 partitions, based on gpt:

/boot: a vfat partition usable for EFI boot (usually ~500MB)
swap: the swap partition (usually ~half RAM)
root: the partition containing the root filesystem

In the the following sections we assume your disk is /dev/sda. If you are using NVMe, your disk might also be /dev/nvme0n1 or similar.

apk add gptfdisk
gdisk /dev/sda
# create new partition table if it does not exist or you want to start clean
# create the partitions

Format partitions

mkfs.vfat /dev/sda1
apk add cryptsetup

# Enter YES and your password twice
cryptsetup luksFormat /dev/sda3

# Create DM device
cryptsetup luksOpen /dev/sda3 rootfs

# Create filesystem
apk add e2fsprogs
mkfs.ext4 /dev/mapper/rootfs

# Mount filesytems
mount /dev/mapper/rootfs /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

Configure initramfs

We need to enable rootfs decryption on boot. For this we need to add cryptsetup into the feature list of /etc/mkinitfs/mkinitfs.conf:

hike:/etc# cat /etc/mkinitfs/mkinitfs.conf
features="ata base ide scsi usb virtio ext4 cryptsetup"

Regenerate the initramfs:

mkinitfs

Configure and install the bootloader

We will be using grub for booting:

apk add grub-efi efibootmgr

Update the /etc/default/grub to contain the cryptroot kernel parameter in the GRUB_CMDLINE_LINUX_DEFAULT variable:

hike:/# cat /etc/default/grub
GRUB_DISTRIBUTOR="Alpine"
GRUB_TIMEOUT=2
GRUB_DISABLE_SUBMENU=y
GRUB_DISABLE_RECOVERY=true
GRUB_CMDLINE_LINUX_DEFAULT="cryptroot=/dev/sda3 cryptdm=root"

Regenerate the grub configuration:

grub-mkconfig -o /mnt/boot/grub/grub.cfg

Verify it has been added correctly:

hike:/# grep crypt /boot/grub/grub.cfg
        linux   /vmlinuz-lts root=UUID=fa67b307-e155-47d8-98a6-4930131b5cd3 ro  modules=sd-mod,usb-storage,ext4 nomodeset quiet rootfstype=ext4 cryptroot=/dev/sda3 cryptdm=root

Install grub:

grub-install --efi-directory /mnt/boot

Install to disk

All changes so far have been done in RAM. Let's persist them:

setup-disk /mnt

Final step

If everything went well so far - it's time to reboot your fully encrypted system. The usual steps like setting up the root password or the hostname have been skipped for the sake brevity.

Enjoy your full encrypted Alpine Linux!

Blocking DHCP servers and router advertisements with nftables

2020-08-27T00:00:00Z

Motivation

Here at ungleich we are providing a variety of hosting services in the Data Center Light. One of the workloads we offer is VM hosting and we need to take some security measures to prevent one customer abusing another customer.

The problem

The virtual machines in our next generation uncloud hosting will be using standard DHCP and IPv6 address assignments. Currently we are still using the OpenNebula contextualisation scripts that read the networking information from an attached ISO.

While this makes it easier to create VM images and VMs behave even more like regular computers, this exposes the VMs to attacks where one customer runs a DHCP server or IPv6 router advertisement daemon and tricks the other VMs into sending traffic to it.

The architecture

VMs are connected to a single shared network in which they get their IP addresses (in uncloud usually only IPv6) and then they can retrieve more information from a metadata server. So the main protection that is required is preventing to trick other customers into using a wrong IP address or route.

Also, if the network is IPv6 only, another customer should not be able to trick someone else into using IPv4.

Fixing it

So the easiest thing to do is to disallow IPv6 router advertisements and IPv4 DHCP server answers. However as all the interfaces are put into one bridge, we will need to filter on bridge and not ip level:

table bridge filter {
    chain prerouting {
        type filter hook prerouting priority 0;
        policy accept;
    }

Next we create a chain to drop the packets we dislike:

    chain drop_ra_dhcp {
        # Blocks: router advertisements, dhcpv6, dhcpv4
        icmpv6 type nd-router-advert drop
        ip6 version 6 udp sport 547 drop
        ip  version 4 udp sport 67 drop
    }

Now the only thing left is to correctly classify the traffic. For this lets take some real world assumptions:

Let's assume the bridge is named br100
Let's assume the upstream interface that should allow RA/DHCP is named vxlan100

Then we can connect the chains together:

table bridge filter {
    chain prerouting {
        type filter hook prerouting priority 0;
        policy accept;

        iifname != vxlan100 meta ibrname br100 jump drop_ra_dhcp
    }

    chain drop_ra_dhcp {
        # Blocks: router advertisements, dhcpv6, dhcpv4
        icmpv6 type nd-router-advert drop
        ip6 version 6 udp sport 547 drop
        ip  version 4 udp sport 67 drop
    }
}

This way we have a very simple filter to prevent router advertisements or dhcp answers to come from customer VMs.

We hope you enjoyed reading it, but if something does not make sense, you can ask on our open chat or consult the nftables reference.

2020 is the year of IPv6

2020-08-07T00:00:00Z

When is IPv6 beginning to matter? When is IPv6 taking off? These are questions that we debate in the IPv6 community at meetings, online and offline.

For us, 2020 has already become the year of IPv6, even way before it is ending.

IPv6 traffic reached more than 33% (1/3) of traffic to Google

According to Google, IPv6 search traffic already exceeds 33%:

The recent growth this year is also attributed to more people working from home, where IPv6 is already more present than in business connections.

33% - this is clearly not yet 100%, however it means that every third search request is made using IPv6.

Some countries passed the 50% IPv6 deployment status

India, Belgium, the US and Malaysia - all of them have passed the 50% IPv6 deployment mark. India even surpassed the 70% mark!

And Greece and Germany are not far from passing the 50% mark (while we in Switzerland only passed the 40% mark...).

In other words: if you are living or travelling to the above countries, you have a good chance of getting IPv6 - and it's growing.

IPv6 community is active and growing

At the moment there are so many cool IPv6 projects and communities around, more active than we have ever seen before. Let us list some projects/communities we are aware of:

IPv6 Buzz by Ed Horley, Scott Hogg, and Tom Coffeen
The IPv6 Chat on Matrix and IRC
The APNIC blog
RIPE labs
IPv6 on Reddit
Afrinic is offering education
The IPv6 blog is a community maintained link collection - add your IPv6 resources there!
#IPv6 on Twitter

.. and probably many more! Did we miss a community? Just let us know about it.

IPv6 hardware/software support improving

For many years we have been watching IPv6 support in (network) hardware and open source software. And while not every everything is fixed, many pain points have been solved. And the great thing is, even if your network equipment does not work with IPv6 nicely, a lot of equipment can now be made IPv6 usable just by flashing OpenWRT.

IPv6 everywhere

You might have heard the claims of our CEO, Nico Schottelius, that you can have IPv6 everywhere.

This year at ungleich we released the VIIRB, the world's smallest IPv6 router.

This device is so easy and so small that you really don't have an excuse for not having IPv6: power it on, plug it into an IPv4 only network: all devices in your network have IPv6 via autoconfiguration.

IPv6 in 2020

So in 2020, you can go out, connect yourself anywhere with IPv6, talk about it with everybody and get your software up and running easily. Oh, and if you think it is just us, we are not the only ones who think the attitude towards IPv6 changed!.

South Pole meets ungleich

2020-06-15T00:00:00Z

Where penguins meet

Do you know where penguins live? Probably all of us assume that they live at the South Pole. That is correct. And it is a really fascinating story how penguins meet. If you haven't seen the movie, we can highly recommend it!

But did you know that penguins also live and meet at other places? For instance in Africa. Or in the case of the penguin companies South Pole and ungleich: in Switzerland.

This blog post is about how penguins and ethical values brought South Pole and ungleich together.

Who is South Pole?

South Pole is a 350-people team of passionate climate experts with its headquarters in Zürich. It provides global climate solutions and develops projects for companies and organizations embarking on their Climate Journey, which includes assessing their carbon footprint and climate risks, green investments, carbon neutrality, and renewable energy.

Started as a handful of people with great ideas back in 2006, now their offices are spread over 18 countries, including Singapore, London, Amsterdam, Sydney, Stockholm, and more. And most importantly: their mascot is a penguin, and the team calls themselves the Penguins.

Sustainability, Switzerland, penguins?! Instant love for us.

We recommend that you take a look at the South Pole blog that covers inspiring projects such as forest conservation, wild animal protection, clean water project for local communities, and many more.

Sustainability at ungleich

A few years ago ungleich started building data centers in the Canton of Glarus in Switzerland. This valley does not only offer a gorgeous view, but was the home of the weaving industry for more than a century. And it's our luck and privilege to build on top of this heritage.

Our focus is on re-using and re-modeling buildings from this time. Many factory halls in Glarus also come with on-site hydropower plants, which allows us to run servers on 100% renewable, locally produced power.

Not only do we re-use old buildings, but with recycled hardware, we were all in for creating a data center that is as sustainable as possible.

When we started this project it was not 100% clear for us what kind of people would be interested in this kind of hosting.

More and more we are realising that how we started and continued our journey has actually brought us the best kind of people. The kind who shares our visions of sustainable technology and understands the value of what we do. So such was the case for team ungleich and team South Pole. We love that those whom we provide our services to have the same mission as we do. Enabling and supporting like-minded people with what we have built has been a really exciting journey for our team.

Continuing our work on sustainability together

South Pole is offering a variety of Sustainability Solutions in all kinds of areas. At ungleich we focus on sustainable hosting in the Data Center Light, the renewable energy-powered data center in the Alps. And we are proud to say that South Pole hosts data in the Data Center Light.

Together we can make a change for a better world.

Join the discussion

If you like to discuss sustainability topics, we invite you to get in touch with us:

ungleich adds a bounty program

2020-04-29T00:00:00Z

TL;DR

At ungleich we love FOSS. If you want to contribute to selected Free and Open Source Software and even get paid for it, checkout the ungleich bounty program.

Introduction

At ungleich we have something like an "infinite task queue". While we do contribute to Free and Open Source Software on daily basis, there are a variety of things we can't do during daily work.

It's Open Source

So because everything we use is Open Source and we live the Open Source spirit, chances are high that somebody will scratch the itch that we have found some time in the future. Because it is Open Source, anyone with the technical skills can actually fix it.

It's about values

Many bounty items will actually list support for IPv6 or fixing things that are necessary for having "good" or clean software. With volunteers contributing to Open Source Software, everyone profits from the changes that you make.

The bounty list

The list of bounties can be found on our Jobs, Hacks and Bounties page. It will be updated regularly with the progress.

If you think you have a project that fits very much the ungleich project, you can also suggest a bounty by writing an email to support at ungleich.ch.

You are also invited to join our open chat, to hang out or to discuss the bounty idea.

Free Video Conference Call For All

2020-04-26T00:00:00Z

A Sustainable Video Conference System

For many people who can work remotely, now is the time to stay home and to move meetings from in person to (video) conference. We have published our remote working stack for those who look for a sustainable solution for digitally continuing their work.

Good news, one more nice thing is added to this list of remote working stack: a free and open source video conference system. Powered by 100% renewable energy, running on a VM at our Swiss data center.

You can try now at talk.ungleich.ch

Deciding what to use

An important note: we’re an open-source, sustainability- focused IT company. We decide which tools to use by these criteria.

open-source
sustainablility
state of the art
usability for people with all backgrounds

Cool thing about Jitsi

The underlying technology for our free video conference call is Jitsi. We love that Jitsi has a really cool architecture - it does not encode or decode the video streams, but only forwards video streams so the client does the encoding and decoding. This means the virtual machine that runs Jitsi doesn't need to be much powerful and can stay fairly small.

Go ahead, use it for free

You can have a video call with multiple parties, you can share screen, and you can share video over it. You can use have a conference call with or without video. No signup is needed, you can just use it on the go. We don't track any of your information: we want to enable people to work from home and stay safe, with open source and renewable energy.

Useful things to know

Our free video conference call is rolled out on March 20th and since then we have collected some feedbacks. Here are some useful infos we got from using it first-hand and from the inputs of our users.

Could show mixed result with video call when used on Firefox. We heard that the great developers of Jitsi are working on this right now and the fix is on its way.
Users have reported that when one call has more than 12 active videos the quality degrades. In such case turning off the video solves the issue.
The video quality varies greatly depending on the cleint computer. Computers with more graphic capability shows better performance.

Call as much as you need, and stay home

Use our free video conference to call the ones you need to talk to. We hope this helps your communication in this extraordinary time we're living in.

If you have any thoughts to share, feel free to get in touch with us at our Twitter or Matrix. Wherever you are, stay safe.

Emacs server the fun way

2020-04-23T00:00:00Z

Today I want to talk about how amazing emacs is. Not because it is the most feature complete operating system out there or because it fully emulates vi/vim. No, because emacs has a very nice feature called emacs server.

What's an emacs server?

If emacs is not an operating system, at least emacs stands for "eight megabytes and constantly swapping", doesn't it? (This is actually from times where 8 megabytes were quite a lot of memory)

So why do people make fun of emacs and how is it related to the emacs server? Emacs, like any other operating system, loads a lot of things at startup. In my case this is initialising org-mode, starting my mail client, calculating my agenda view to tell me what to do today.

An emacs server creates a special emacs process that listens on a socket for connecting to it. This way the initialisation is already done before you connect to it and all configurations are already loaded. This is the actual "slow" part of emacs. And is a bit similar to starting python, which also needs to load its libraries at start.

With the emacs server running, you can connect to it using the emacsclient program.

As a matter of fact, rxvt-unicode also knows about a server mode (checkout the manpage, look for urxvtd -q -f -o). For rxvt-unicode, you'd use urxvtc to connect to it. So quite simlar.

What is so cool about the emacs server?

Saving a lot of response time and making working with emacs feel much faster is the obvious advantage. However, there is a much bigger one:

With the emacs server, you can connect to it from the terminal and X Window. Because the emacs server also manages the buffers ("open files" for non-emacs users), you can view the same open file from the terminal or an x window.

Turning the notebook into a server

As you might know, we at ungleich are pretty much into IPv6. So all of our devices are generally speaking world-wide reachable. Our work notebooks are no exception from that. In fact, most notebooks even have their own /48 IPv6 network assigned via VPN.

So if I am away from my notebook, but need to check my open (and potentially unsaved) notes or view my emails, I can use any other computer, ssh to my notebook and type emacslient -nw in the terminal.

While my regular emacs is running as an X11 window, I can select, display and work in all buffers that I have previously opened in the emacs server. In the terminal, on a remote computer.

How to configure your system to use the emacs server

In my case I start the emacs server when I start X11 in my .xinitrc:

eval $(ssh-agent)
...
urxvtd -q -f -o
emacs --daemon
...

Instead of running emacs --daemon, you can also use M-x server-start in a running emacs process.

And because I always want to have my mail client open, just after I start i3, I have i3 launch the following command:

ssh-add </dev/null && emacsclient -c -e '(mu4e)'

The connection to my mailserver is tunneled via SSH to prevent security issues from using SSL/TLS. Thus I need to add all my ssh keys to the ssh-agent, before starting my mail client.

To actually open a new emacs windown (aka "frame" in emacs speech), I use the following configuration in my ~/.i3/config:

bindsym $mod+Tab    exec emacsclient -c

How does this look like?

Below you find a screenshot of my writing this article. The upper window is the X11 window, the lower window is a terminal window (they happen to be configured to have the same nice background colour).

Updates

2020-04-23, 17:23

MutoShack pointed out on reddit that using

emacsclient -a "" -c

is actually smarter than just using

emacsclient -c,

because it will start the daemon if it is not already running. No day that you don't learn something!

Deutschsprachiger Coronavirus (COVID19) Chat

2020-03-22T00:00:00Z

Wie es funktioniert in Kürze

Gehe auf https://account.ungleich.ch und erzeuge einen neuen Account
Schaue in Deine Inbox, verifiziere Deine E-Mail
Gehe auf https://matrix.ungleich.ch und melde Dich mit Deinem Benutzernamen an
In dem Textfeld gib /join #covid-19-de:ungleich.ch ein
Nun kannst Du Dich mit anderen austauschen, auch wenn Du zuhause oder isoliert bist

Kontakt trotz sozialer Distanz

In Zeiten des Coronavirus ist es wichtig, dass wir Abstand zueinander halten und uns nicht gegenseitig infizieren, damit unser Gesundheitssystem nicht überlastet.

Damit wir in der Isolation trotzdem Kontakt miteinander haben können, stellen den offenen Kanal #covid-19-de:ungleich.ch zum Austausch zur Verfügung.

Warum nicht Whatsapp, Facebook oder ähnliches?

Es gibt 1 grosses Probleme bei Whatsapp, Facebook, Telegram, Signal, Threema und co.:

Es sind alles geschlossene Systeme. Jemand der nur auf Whatsapp ist, kann nicht mit jemanden von Telegram kommunizieren.

Der Matrix-Chat ist anders: er ist offen und man kann sich mit Teilnehmern von verschiedenen Matrix-Systemen unterhalten. Und man kann Gruppen von anderen Systemen wie Telegram oder Whatsapp einbinden.

Wie es funktioniert

Du kannst Dir einen Account auf https://account.ungleich.ch anlegen. Danach bekommst Du eine E-Mail, die Du bestätigen musst. Nachdem der Account erstellt ist, kannst Du Dich auf https://matrix.ungleich.ch anmelden. Um dann mit anderen zum Thema Coronavirus zu sprechen, tippe /join #covid-19-de:ungleich.ch ein.

Für Englischsprachige Personen gibt es bereits den internationalen Chat #covid-19:ungleich.ch, mit einer englischen Anleitung.

Eigene Chat-Server für Firmen

Der Chat ist kostenlos und öffentlich verfügbar. Wenn Du einen eigenen Chat brauchst für Deine Organisation, kannst Du diesen auf der Hosted Matrix Chat Webseite bestellen. Dies ermöglicht Dir, selber Deine Benutzer zu verwalten.

Falls Du jedoch nur mit Deinen Freunden und Familie Dich unterhalten willst, kannst Du einfach kostenlos den Dienst auf https://matrix.ungleich.ch benutzen.

Coronavirus

Es liegt an uns allen, die Folgen des Coronavirus zu begrenzen. Dies passiert nur, wenn wir alle Vorsicht walten lassen und Abstand zu anderen halten und die Hände häufig waschen.

Macht mit und helft allen anderen. Danke fürs Lesen!

코로나바이러스 챗 (한국어)

2020-03-22T00:00:00Z

밖에 안나가고 바깥 세상과 디지털로 연결하기

https://account.ungleich.ch 에서 계정 만들기 (무료)
계정 생성 확인 이메일 받기
https://matrix.ungleich.ch 로 가서 새로 만든 계정으로 로그인하기
/join #covid-19-kr:ungleich.ch 바이러스와 관련된 얘기하기
사람들과 소통하세요. 당신은 혼자가 아닙니다.
원하면 새로운 챗방을 만들수 있습니다
계속 집에 계세요.

코로나바이러스, 재택근무, 정신줄 챙기기

코로나는 많은 사람들의 생각과 행동, 근무방식을 바꾸고 있습니다. 자발적으로든 강제적으로든 재택근무하게되신 여러분, 그래도 세상과 소통하고 싶으실거예요. 동료들 또는 다른 나라에서 같은 처치에 처한 분들과요.

집에 계세요. 하지만 우리는 연결되어 있죠.

Chat Francophone à propos du Coronavirus (COVID-19)

2020-03-22T00:00:00Z

Version courte:

Va à l'adresse https://account.ungleich.ch et créé-toi un nouveau compte
Vérifie ta messagerie (y.c. le spam ;-) et valide ton adresse e-mail
Va à l'adresse https://matrix.ungleich.ch et connecte-toi avec ton nouveau compte
Dans le champ texte, copie /join #covid-19-fr:ungleich.ch
Tu peux désormais échanger avec d'autres francophones depuis chez toi ou en isolement

Gardons le contact malgré la distanciation sociale

En cette période de pandémie, il est important que nous gardions une certaine distance entre personnes et que nous ne nous infections pas mutuellement, afin de ne pas surcharger nos systèmes de santé. Afin de maintenir le contact malgré l'isolement, nous mettons à disposition le canal ouvert #covid-19-fr:ungleich.ch pour les échanges d'idées, d'informations et simplement de contact social

Pourquoi pas Whatsapp, Facebook ou similaire?

Il y a un gros problème avec Whatsapp, Facebook, Telegram, Signal, Threema, etc.: Ce sont tous des systèmes fermés. Quelqu'un qui n'est que sur Whatsapp ne peut pas communiquer avec quelqu'un qui n'utilise que Telegram. Le chat Matrix est construit différemment: il est ouvert et accessible à des utilisateurs de différents systèmes Matrix. Il est aussi possible d'intégrer des groupes d'autres systèmes comme Telegram ou Whatsapp.

Comment ça fonctionne?

Tu peux créer ton propre compte sur https://account.ungleich.ch. Tu recevras un email pour vérifier et valider ton compte, que tu devras approuver. Une fois ceci fait, tu pourras te connecter sur https://matrix.ungleich.ch. Pour rejoindre le groupe francophone consacré au nouveau coronavirus, tape simplement /join #covid-19-fr:ungleich.ch Il y a également un groupe germanophone: #covid-19-de:ungleich.ch avec son manuel: https://ungleich.ch/u/blog/deutschsprachiger-coronavirus-chat/. Et un anglophone: #covid-19:ungleich.ch et son manuel: https://ungleich.ch/u/blog/work-from-home-work-remote-with-matrix/

Avoir son propre serveur de chat pour une entreprise

Ce chat est gratuit et publiquement disponible. Si tu as besoin de ton propre système pour ton organisation, tu peux le commander à cette address: https://ungleich.ch/u/products/hosted-matrix-chat/. Ceci te permet de gérer toi-même tes utilisateurs. Si tu veux simplement discuter avec tes amis et ta famille, tu peux simplement utiliser le service gratuit https://matrix.ungleich.ch

Coronavirus

C'est à nous tous de limiter les conséquences du Coronavirus. Ceci ne se fera que si nous faisons tous attention, que nous gardons nos distances et que nous nous lavons les mains régulièrement. Participez et aidez votre prochain! Merci d'avoir lu jusqu'ici!

List of Coronavirus (COVID19) Chats

2020-03-22T00:00:00Z

Instead of meeting in person, you can chat with others about your coronavirus thoughts. It is available in the following languages:

Auf Deutsch #covid-19-de:ungleich.ch
In English #covid-19:ungleich.ch
En Français #covid-19-fr:ungleich.ch
한국어 #covid-19-kr:ungleich.ch

All channels are available via the open Matrix chat and can be joined with an account from any Matrix server. Details can be found in the links.

Remote Working with Open Source

2020-03-16T00:00:00Z

Working at the time of Covid19

So looks like 2020 is the year of remote working for a whole lot of people. Hey, welcome to the club! Our team has been working remotely since 2013, and now we are distributed in 5 different time zones. For us remote working has been a lot of fun, all you need is the right setup that enables everybody to be connected and be productive.

We obviously couldn't do it without the digital infrastructure. Lately we've been getting a lot of requests for sharing tips for which tool or software to use for remote working, and we thought it'd be a good idea to share a practical list of tech stack we use in our daily business. So, here we go: remote working stack chez ungleich.

How we decide what to use

Before we continue, an important disclaimer: we’re an open-source, sustainability- focused IT company. So we decide which tools to use by these criteria.

open-source
sustainablility
state of the art
usability for people with all backgrounds

1. Chat: Zero carbon chat - Mattermost and Matrix

You need a chat system when working remotely with your team, and you need it to be secure and hassle-free. We used to be on Slack at the very beginning, but we did not like the fact that Slack is hosted with AWS: from our point of view AWS datacenters are not sustainable compared to our own renewable energy datacenter in Switzerland, so we moved away from Slack into hosting our own chat.

Now we have two chat systems, Mattermost and Matrix. Mattermost offers an extremely easy and smooth team communication for people with all kinds of backgrounds. Most of our team talk happens on this platform. Our chat is open for the public and anybody can join with their email and have a conversation with us.

Zero Carbon Matrix, on the other hand, offers end-to-end encryption and federation, which makes it very attractive for people with high sensitivity for privacy and security. It also allows us to be more open. Matrix works nicely with video and audio calls as well, suitable for the secure conference call with different teams.

2. Cloud storage - Nextcloud

We are using hosted Nextcloud for data storage and sharing. It supports cloud storage, file sharing between teams and real-time collaboration for document editing. It's pretty easy and we like to save data in Switzerland and not somewhere we don't trust how the authorities operate.

3. Project management - Redmine

Redmine is our longest standing project management tool since the beginning of our company. It supports calendar, ticket creating, issue tracking, wiki, roadmap and more.

Redmine is just very handy for managing multiple projects and subprojects, and giving access to different people can be done on a role basis. Our whole data center infrastructure is actually in redmine & we open it for the public, so you can take a look to get an impression.

4. Code hosting - Gitlab

We used GitHub before, but we decided to move to our own Gitlab in 2018 due to GitHub's lack of #IPv6 support and some other issues. We are very happy with our new home, and that's why we're hosting Gitlab for our customers too.

A decision with sustainability goes a long way

As we disclosed at the beginning, sustainability is an important factor for us so we run all of the above on our servers with renewable energy. Many of you who are new to remote working might have to make decisions in a hurry, but try to remember that decisions with sustainability in mind saves a lot of pain in the long run, and also saves our environment.

Questions?

It can be overwhelming when you need to decide what to use for a new style of work: you'll have to use it every day and you want to make the right decision. Our approach is that we know where are values stand such as decentralisation, open-source, sustainability, and we try to make our decisions aligned to our values. For us working remotely with open source is fun and exciting, it gives so much freedom to everybody involved. If you have any questions on the above stack, our remote-working team is around to give you answers. ;) Stay healthy, and happy remote working to you all!

Remote Working with Matrix in covid-19 times

2020-03-15T00:00:00Z

TL;DR

Stay isolated in real life, stay connected digitally.

Go to https://account.ungleich.ch, create an account, it's free
Verify your account by email
Go to https://matrix.ungleich.ch, login with your new account
Type /join #covid-19:ungleich.ch if you want to talk about topics related to the virus.
Talk to people. You are not alone.
Create your own rooms as needed (as in above channel if you need help)
Stay home

Coronavirus, remote working and how to not go crazy

The covid-19 virus forces a lot of us to think differently, behave differently and work differently. Whether you have to work from home or whether you do it voluntarily, you might want to stay in touch with other people. With your teammates, colleagues or other people.

Isolate in real life, stay connected digitally

One of the most important things to do at the moment is to not further spread the virus. Here in Switzerland, we try to reduce the risk by social distancing. No shaking hands, coughing and sneezing into your elbow and in general to stay away from other people.

If you are not used to staying alone or being isolated, this can be a tricky thing to do. Luckily, we do have the Internet that allows us to talk with each other, without causing a danger to each other.

While there are a variety of tools out there to stay in touch with each other, we propose to use the Matrix chat for a variety of reasons:

Matrix is federated. Federated = open to talk to others

Matrix is federated. As opposed to Whatsapp, Telegram, Hangouts, Skype or Slack, Matrix is open to connect to others. Let me guide you through how it differs to other systems:

If you create an account on https://account.ungleich.ch and then go to https://matrix.ungleich.ch afterwards, you can talk to anyone in the world, who is also using Matrix. Not the one from ungleich, but maybe the one from https://matrix.org. Or from somebody completely different.

Matrix is not a closed system. If you decide that you want to write to somebody on a different server, you can do so.

Matrix also allows you to call other people or have video conferences.

A Matrix channel dedicated to COVID-19

So coronavirus. Social distancing. We created a channel dedicated to the coronavirus to discuss the topic. If you created an account above and joined https://matrix.ungleich.ch, you can join the channel by typing /join #covid-19:ungleich.ch or use the matrix.to service.

The covid-19 channel is open for everyone. You can discuss how you feel, whether you are affected, infected, bored or ask questions about how to behave. Beware though, we are not a medical team. This is more the type of alcoholics anonymous style, not a digital hospital.

There is now also a German speaking channel available.

Your own Matrix server

If you are newly into remote work and you want or need your own Matrix server, this is something that we provide. You can checkout the Hosted Matrix Chat for details. It allows you to manage your own chat rooms and to control whom to invite to what.

If you just need a place to talk with your friends, you can do so freely on https://matrix.ungleich.ch.

Future of the coronavirus

No one of us knows at the moment how things will continue with the covid-19 virus, but we wish everyone to stay safe. And hopefully above chat helps some of us to not stay too alone, even if we are alone at home.

Stay safe!

What I deployed on IPv6 this week

2020-02-26T00:00:00Z

IPv6 is growing and everyone in the IPv6 community contributing to it. This article is dedicated to show what you have done with IPv6 or where you have enabled IPv6 this week.

Your story here

Do you have something interesting to share? Either send your short story to ipv6 at ungleich.ch with the subject What I deployed on IPv6 this week or submit a merge request to the ungleich-staticcms repo (account can be created on https://account.ungleich.ch).

Anything IPv6 related is welcome!

2020-02-24 - 2020-03-01

Thomas Schäfer installedd a ripe atlas probe with IPv6 only
Nico Schottelius deployed a prototype of uncloud v2 on the IPv6 only host manager.place7.ungleich.ch that serves as a test bed for migrating the OpenNebula based virtualisation to uncloud.

More of it

Feel free to join the discussion on the IPv6.chat.

Proying IPv4 traffic via the ungleich VPN

2020-02-18T00:00:00Z

We have been offering an IPv6-capable VPN alongside our IPv6-only VPS hosting for a while in order to bring IPv6 connectivity to customers stuck in the IPv4 world. The service also allows you to reach the IPv6-enabled side of global Internet but was not able to connect to IPv4-only services (such as github!), which can be painful depending on your use-case.

This shortcoming is no more since we recently deployed two DNS64 resolvers available to any VPN user. They will generate a synthetic IPv6 address for domains lacking an AAAA (i.e. IPv6) DNS record, which will in turn be routed via our NAT64 gateway. You only have to configure 2a0a:e5c0:2:12:0:f0ff:fea9:c451 and 2a0a:e5c0:2:12:0:f0ff:fea9:c45d as DNS servers when you are connected to the VPN: all the details and instructions are available on our wiki, although it boils down to two lines in your wireguard configuration.

The above means that ungleich now provides a fully-fledged VPN! Note, however, that direct IPv4 queries (i.e. requests 'bypassing' DNS resolution) won't be routed though the VPN. Full isolation can be achieved using network namespaces as described in the wireguard documentation. Feel free to join our chat to discuss such (non-trivial) setup in details!

How to route IPv4 via IPv6

2020-02-10T00:00:00Z

Imagine the following: you are running an IPv6 only network. And now someone asks you to pass IPv4 traffic through it, without tunneling it. Was sounds crazy at first, is actually quite feasible.

A short routing recap

Routers have routing tables. The routing tables basically say "if you receive a packet for this host, send it to that router".

The important thing about this process is that the information on where to send it to, is not in the packet.

How to send IPv4 packets via IPv6

Because the next hop is not written into the IPv4 packet, the router is free to forward the packet via any method it thinks is the best. And if that happens to be IPv6 - well, it will forward the IPv4 packet via an IPv6 neighbour.

A practical example!

In the IPv6 only coworking network in the Digital Chalet, I can add an IPv4 default route via the IPv6 router:

[root@diamond ~]# ip route add 0/0 nexthop via inet6 fe80::21b:21ff:febb:6934 dev wlp0s20f3
[root@diamond ~]# ip r
default via inet6 fe80::21b:21ff:febb:6934 dev wlp0s20f3
[root@diamond ~]#

Now to be able to actually transmit IPv4 packets, I do need a source IPv4 address. In the current network I can use an address in the unused 10.0.8.0/22 network, however I'll add it with a /32 mask to make it clear that there is no interface local route applied:

[root@diamond ~]# ip addr add 10.0.8.42/32 dev wlp0s20f3
[root@diamond ~]# ip r
default via inet6 fe80::21b:21ff:febb:6934 dev wlp0s20f3
[root@diamond ~]# ip a sh dev wlp0s20f3
2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:ee:9a:54:c3:bf brd ff:ff:ff:ff:ff:ff
    inet 10.0.8.42/32 scope global wlp0s20f3
       valid_lft forever preferred_lft forever
    inet6 2a0a:e5c0:0:4:c6ea:b1a8:ec14:6f35/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86400sec preferred_lft 14400sec
    inet6 fe80::3b98:cb58:ed02:c25/64 scope link
       valid_lft forever preferred_lft forever
[root@diamond ~]#

And I can indeed ping another IPv4 address, routed via IPv6!

[root@diamond ~]# ping -4 10.0.8.3
PING 10.0.8.3 (10.0.8.3) 56(84) bytes of data.
64 bytes from 10.0.8.3: icmp_seq=1 ttl=64 time=2.37 ms
^C
--- 10.0.8.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.365/2.365/2.365/0.000 ms
[root@diamond ~]#

Why?

Why would anyone want to do this? It's quite easy: with this you can route an IPv4 address to an IPv6 only host. This enables IPv6 only resources to create and send IPv4 packets, even if they don't have IPv4 routes.

Do it yourself

If you don't believe us that it is possible, you can test it yourself on IPv6 only VMs on IPv6OnlyHosting.com.