Dom Goodwin

Don’t just say ‘hello’ to me

dom@domgoodwin.dev (Dom Goodwin) — Wed, 25 Jan 2023 19:06:09 +0000

I was recently in a 1-1 with my new EM and was asked what annoys me at work. The one thing that immediately popped to mind was:

‘I hate it when someone just messages me “Hello” or “Hi, how are you” when they message me for something on Slack’

It became immediately apparent that, without proper context, this could be taken the wrong way. So I thought I’d explain my thoughts here.

It has annoyed me since the advent of instant messaging at work, but as more people now work remotely the problem has only got worse.

At a high level, the issue is this: instant messaging is asynchronous, unlike a verbal conversation, each participant can be doing something else at the beginning of the chat and might continue doing so for the duration of the conversation.

With a phone call, communication is instantaneous, so the time for pleasantries is tiny compared to the actual discussion of a problem. When it’s via chat there’s delay, even if both parties are watching the chat, waiting to respond as soon as possible, they still take time to type and find the right emojis.

Let’s use a sequence diagram to show what I mean:

Bob and Alice talk in serial

This is a pretty typical conversation, even ignoring Bob’s rudeness of not asking after Alice. Alice needs to ask Bob a question, but starts the conversation as you would in-person or by phone. If we think of humans as single core processors, both are basically ‘blocked’ for the entire duration, unable to do anything else until this conversation is over.

This gets worse if you factor delays into the responses:

Bob and Alice talk in serial with delays

Now obviously this is a worst case scenario, both parties could be doing other things whilst waiting for responses, but that would also introduce more interruptions to their work. Adding a context switch every time someone sends a message would slow down whatever work they were doing, as an example:

Bob and Alice talk in serial with context switch delays

Here, Bob is trying to complete his work whilst also responding to Alice. Each time he responds then returns to his work. However, this means he needs to context switch and remember what he was working on before the interruption.

Now, how could this be improved:

Bob and Alice exchange a single question

Alice has a question, she messages Bob the question but then go back to their work (if possible).

Bob receives the question, thinks through an answer and sends it back to Alice who is then unblocked.

Alternatively, Bob is working on a tough problem so doesn’t see the message until 30 minutes later, he then responds and Alice who was looking at a different problem can then get her answer.

Regardless of how long a response takes, both parties spend the same time on the communication. Compare that to earlier examples, even if Bob context switched with every message, the time consumed with the same outcome (Alice has her answer) is far greater.

ℹ️ A lot of this boils down to just sending someone https://www.nohello.com/ which explains this a lot more succinctly than I could.

Reducing our NAT Gateway cost with private networking between AWS and GCP

dom@domgoodwin.dev (Dom Goodwin) — Sun, 15 Jan 2023 10:06:09 +0000

This post is hosted on the Monzo Blog:

Reducing our NAT Gateway cost with private networking between AWS and GCP

Kubernetes Networking #3: Ingress

dom@domgoodwin.dev (Dom Goodwin) — Fri, 22 Jul 2022 18:06:09 +0000

Kubernetes Ingress provide a way to expose HTTP(S) routes into your pods from a centralized controller and load balancer. It means you can define how the network traffic can get to your pod inside a Kubernetes resource and a controller hosted in the cluster takes care of the implementation.

In a world without Kubernetes, you might have a reverse proxy deployed where you write the configuration file to detail how to expose your applications in the backend. An ingress controller takes that reverse proxy and automatically generates the config based on the Ingress Kubernetes resources you create. You don’t have to create a new LoadBalancer service and cloud load balancer for each service you want to expose.

In this guide, we’ll use the Kubernetes maintained ingress controller, ingress-nginx, which uses nginx as it’s reverse proxy. It’s important to note, this isn’t the nginx-ingress-controller from NGINX themselves which has hooks into NGINX plus and other paid offerings. The docs for ingress-nginx can be found here.

🤷 Why not just LoadBalancer?

Kubernetes provides a way for a cloud provisioned load balancer (an AWS elastic load balancer for instance) to be deployed by the control plane, see my services post for more info. We can use these to expose a single service (and collection of pods) from a load balancer. But what happens when you have 50 services deployed onto your cluster? Would you have 50 load balancers for them all? That’s ~$810 in standing charges every month not including paying for the traffic that goes through them.

This is where an ingress controller comes in. An Ingress controller is a centralized place for traffic to enter your cluster via a single load balancer which then is routed to your backend services automatically. This is all based on Ingress resources deployed on the cluster, through the reverse proxy. It also introduces a place to add some control over your traffic routing. kube-proxy will default (when in iptables mode) to TCP load balancing traffic randomly, the traffic coming in via a load balancer goes to any pods. With the controls you can configure affinities, round-robin routing, rate limits, CORS etc. to really shape the traffic going to your pods centrally.

These diagrams compare the path a request would take with/without an ingress controller exposing your pod:

Without: Application Service LoadBalancer → NodePort → Randomly to one of your pods

With: Ingress-controller service LoadBalancer → NodePort → Randomly to an ingress-controller pod → To one of your application pods via ClusterIP

There are some tradeoffs to using an ingress controller:

This introduces an extra hop for your network requests to reach your pod. If you have high throughput - and low latency is important - this could be a consideration
In the same vein, the ingress controller pods also use resources. If you have the ingress controller doing a lot of TLS termination or handling high load the resource usage will increase significantly
You can only really use this for (mostly) Layer 7 traffic, TCP or UDP cannot directly be exposed this way

🛠️ So what does an ingress controller actually do?

At a high level you have the following components:

Kubernetes resources:
- Service: type LoadBalancer, meaning your ingress-controller pods are fronted by a cloud load balancer and exposed outside the cluster
- Deployment: Ingress Controller itself (see components below for a breakdown)
- (sometimes)Deployment: default backend, this is where the ingress controller routes traffic to when there isn’t a matching host/path rule for the traffic coming in
Components of the ingress controller:
- Reverse proxy: This is what actually routes the traffic coming in to downstream services/pods. If you installed nginx on a service and ran it it’s this basically
- Config generator: This continuously gets the state of Ingress resources in the Kubernetes cluster (think of it as spamming kubectl get ingress -A every X-seconds) and uses that to generate config that the reverse proxy (nginx in this case) understands

The reverse proxy continually has it’s config updated by the config generator and then traffic coming into the cluster via the Loadbalancer service hits the reverse proxy and is routed to the proper backend service/pods.

🔒 What about SSL?

For HTTPS traffic, we can configure the ingress-controller to terminate the SSL and provide the certs as needed.

When creating an ingress resource you can specify a Kubernetes Secret of type tls and the ingress-controller will use that secret to terminate the SSL of the request at the reverse proxy.

This could also be coupled with a tool like cert-manager to dynamically issue and renew certificates for you.

🎛️ Load balancer/Reverse proxy controls

As mentioned before, by having your pods traffic route via a reverse proxy in the cluster you gain more control over the routing and connection. Some useful examples for ingress-nginx are:

Authentication: Setup basic or digest access authentication on routes
- nginx.ingress.kubernetes.io/auth-type: basic
Session affinity: Allows for things like sticky sessions, keeping a user going to the same backend pod or use cookie affinity to apply a SameSite sticky cookie
- nginx.ingress.kubernetes.io/affinity: sticky
SSL passthrough: Instead of terminating TLS at the load balancer you can let the backend handle it directly
- nginx.ingress.kubernetes.io/ssl-passthrough: "true"
Redirections: Both permanent and temporary
- nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com
Security controls
- SSL ciphers can be configured controlling which SSL ciphers you support
  - nginx.ingress.kubernetes.io/ssl-ciphers: "ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
- CORS rules to configure the cross-original resource sharing rules
  - nginx.ingress.kubernetes.io/enable-cors: "true"

A lot of these controls can be configured either as annotations directly on Ingress resources affecting only those specific routes, or can be provided in the Configmap to be applied at a global level.

🤹‍♂️ Can I have multiple ingress-controllers?

Yes. Let’s say you want to be able to accept traffic from the public internet as well as internally from inside your private network.

We would:

Deploy two sets of ingress-controllers, ingress-public and ingress-private
Configure them to be public and private. This is done usually by the annotations given to the different Service resources (service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" for instance)
Setup the Ingress classes
- This is a relatively new resource, it used to be managed by an annotation kubernetes.io/ingress.class which is being deprecated
When creating Ingress resources you can target the IngressClass to specify which controller should setup routing for your traffic

📮 What about hostnames?

Ingress resources in Kubernetes allow you to specify both the Paths and the Host values to route traffic for.

To have your ingress controller exposed via a friendly DNS address you’d need to point a record of one to your load balancers DNS address. In AWS, this would be creating a Route53 A record aliased to your ELBs address. This would mean anything going to friendly-address.domain.com would point to your cloud-provisioned ELB and then onto your reverse proxy pods from there.

Then any routes off this hostname would be directed to your specified pods, ie.:

`friendly-address.domain.com/app-a` → `svc/app-a` in the cluster
`friendly-address.domain.com/app-b` → `svc/app-b` in the cluster and so on

But what if you wanted to have multiple different hostnames, maybe 1 per application.

`app-a.friendly-address.domain.com/` → `svc/app-a`
`app-b.friendly-address.domain.com/` → `svc/app-b`

These would mean adding a new Route53 entry manually every time you want to deploy a new application. It takes away from the benefits of defining Ingress as a Kubernetes resource if you need to go click-ops or Terraform a new host each time.

That’s where a tool like external-dns comes in.

external-dns

external-dns allows you to automatically configure your DNS services based on your Kubernetes Ingress resources (and services!).

As an example, if you have deployed external-dns and wanted to setup a new app-c.friendly-address.domain.com to point to your service all you would need to do is create an Ingress resource with the Host field set to app-c.friendly-address.domain.com and external-dns would provision you an A record pointing to your load balancer.

🍵 Bringing it all together

To give an example of all of this in use, lets build resources for the following requirements:

We have 3 applications: alpha beta and gamma running on our cluster
We want alpha and beta exposed via the same Hostname but with separate routes:
- public.example.com/alpha and public.example.com/beta
We want gamma to have it’s own hostname as it’s a separate, unrelated application
- gamma.public.example.com
We have the TLS key and cert for public.example.com and want the ingress-controller to terminate TLS for this host
gamma handles it’s own TLS so we shouldn’t terminate it at the reverse proxy,

We already have:

A Kubernetes cluster running in AWS
Deployed external-dns onto our cluster configured for AWS Route 53, [link]
Deployed ingress-nginx onto our cluster configured for AWS, [link]. Ingress class is set to default

Feel free to try these steps above for yourself the implementation and breakdown of what we’re doing come below though so spoiler alert.

Create the TLS secret for our alpha/beta endpoint

kubectl create secret tls public \
  --cert=tls.cert \
  --key=tls.key

YAML for the Ingress of alpha and beta

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: alpha-beta
spec:
  ingressClassName: default
  tls:
  - hosts:
      - public.example.com
    secretName: public
  rules:
  - host: public.example.com
    http:
      paths:
      - path: /alpha
        pathType: Prefix
        backend:
          service:
            name: alpha
            port:
              number: 8080
      - path: /beta
        pathType: Prefix
        backend:
          service:
            name: beta
            port:
              number: 8080

YAML for the Ingress of gamma

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gamma
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  ingressClassName: default
  rules:
  - host: gamma.public.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: gamma
            port:
              number: 8080

☝ SSL Passthrough also needs to be enabled on the ingress-nginx deployment with the flag --enable-ssl-passthrough

What happens with these resources?

ingress-nginx:
- Constantly checks for new/updated Ingress resources
- Detects alpha-beta:
  - Adds nginx config to route to the services
    - Can be checked with: kubectl exec -it <ingress-controller-pod-name> -- cat /etc/nginx/nginx.conf
    - This config includes the secret mentioned to be able to terminate TLS
  - Updates the Ingress resource Address field with the DNS of the Cloud load balancer the controller knows is pointing at it
  - curl https://public.example.com/alpha -v shows it routing correctly and the cert as expected
  - curl https://public.example.com/wrong -v shows the default backend
- Detects gamma:
  - Adds nginx config to route to the services
    - Again, can be checked with: kubectl exec -it <ingress-controller-pod-name> -- cat /etc/nginx/nginx.conf
    - This config will show the ssl_passthrough is set to not handle any TLS
  - Updates the Ingress resource Address field with the DNS of the Cloud load balancer that the controller knows is pointing at it.
  - curl https://gamma.public.example.com -v shows traffic routing correctly and with our in-application TLS handling
external-dns:
- Constantly checking for new/updated Ingress resources
- Sees the two new resources but with empty Address fields - needs these to be set before it can point the specified host to the address the service sits behind.
- Address field is now updated for alpha-beta:
  - Checks if the hostname has already been created
    - external-dns defaults to using txt records in the chosen DNS service to persist information about the domains it manages
  - Creates an A record pointing public.example.com to the value of Address in the ingress
- Address field is now updated for gamma
  - This does the same as alpha-beta but for gamma.public.example.com

🦦 Other controllers are available

Other ingress controllers are available too, notable/interesting ones include:

AWS Load balancer controller can manage ALBs out of cluster for ingress resources and NLBs for service resources
Traefix ingress controller an ingress controller using the Kubernetes native edge router
Ambassador uses Envoy to act as an ingress controller and API gateway

The benefit of having all of your configuration directly in Kubernetes as an Ingress resource comes from every compliant ingress-controllers reading and interacting with Ingress in the same way. The only change between them would be the implementation specific things you control via annotations, the general routes and hosts work regardless of your choice.

Kubernetes Networking #2: Services

dom@domgoodwin.dev (Dom Goodwin) — Thu, 21 Apr 2022 10:06:09 +0000

Kubernetes services are an essential resource everyone uses, but what are all the different types and how do they work under the hood?

In Kubernetes pods are ephemeral (or cattle), you should expect them to be killed, scaled and replaced whenever. So how do you talk to a set of pods which might have changed the next second? A service.

Services sit in front of your application and mean you only have to call the service address and magic Kubernetes components will get that request to the right Pod(s). How does a service know which pods it can talk to?

Selectors

Services map to pods via label selectors. You have a pod with label app=etizer and then create a service with .spec.selector for the same and that service will forward traffic to any Ready pods matching that label.

Behind the scenes, a service resource also creates an Endpoints resource which maintains a list of Pod IPs based on your service selector. It’s worth noting here, the readyiness checks you implement for your Pods are what controls whether the IP goes into this Endpoints resource and allow traffic.

The exception here is selector-less services, these don’t have a label selector so won’t also have an Endpoint resource. To use these you also need to create an endpoint resource of the same name as your service, you can add any IPs and ports you want to this resource. This is a great way to map a service to something not in your cluster so your pods can talk to it as if it was a service.

But what are the different types of services and how do they work?

Service Types

ClusterIP

When you create a ClusterIP type service you’ll will get:

A virtual IP for the service
- This is within the --service-cluster-ip-range your api-server has been configured with
- kube-proxy is responsible for this virtual IP routing to one of your Pod IPs
- This virtual IP will forward requests to one of your pod IPs based on the Endpoints resource associated with your service.
A local cluster DNS address, eg. <service-name>.<namespace>.svc.cluster.local
- This is resolved by your kube-dns (CoreDNS) to your virtual service IP
- This is why your service name must be RFC 1035 compliant

The normal way of utilising this would be giving your application the service DNS address so it can talk to another application running on the cluster. This DNS address resolves to the virtual IP and then the configuration kube-proxy controls handles that IP resolving to actual pod IP addresses. From there your CNI will route your request using the pod IP.

The ClusterIP service type is ✨special✨, whilst you use it as a way to expose your service internally within the cluster, it’s also used by types NodePort and LoadBalancer (by extension of LoadBalancer using NodePort) as part of their implementations.

Just like a Matryoshka doll

<service-ip>:<service-port> → <pod-ip>:<pod-port>

NodePort

A NodePort type service opens up a port on every node in your cluster, this port then proxies requests to your service’s virtual IP address.

The port range defaults to 30000-32767, interestingly enough just below the Linux kernel default ephemeral port range. It can also be configured with the flag --service-node-port-range if you need these ports for some use case.

At a high level a NodePort is a single rule configured by kube-proxy onto every node, to forward requests to an underlying ClusterIP service.

<node-ip>:<node-port> → <service-ip>:<service-port> → <pod-ip>:<pod-port>

LoadBalancer

A LoadBalancer type service only really works in a cloud environment. When created, asyncronously, the cloud-controller-manager will provision you a load balancer which directs traffic to your pods.

The implementation of the different cloud load balancers varys wildly. It used to be different clouds were implemented in-tree, committing directly to kubernetes but now it must be done via out-of-tree implementations, eg. cloud-provider-aws.

Cloud implementations aside, once you have a cloud load balancer it works by pointing to a NodePort service running. This means that you have a load balancer with every Kubernetes worker node as a listener, forwarding traffic to any nodes on a particular port. From there it follows the standard NodePort → ClusterIP route.

If you didn’t want to write a whole cloud control manager to expose your application on your own servers, a NodePort service with that port pointed to from your existing load balancer would do the trick.

<load-balancer-ip>:<lb-port> -> <node-ip>:<node-port> → <service-ip>:service-port> → <pod-ip>:<pod-port>

ExternalName

ExternalName services are the odd one out. Instead of using good ol’ ClusterIP and getting a virtual IP, they create a CNAME record to map your service DNS to another DNS address.

As an example use case, lets say you have different environment databases, instead of service-owners changing the DNS depending on where they deploy their pods to you create a ExternalName service:

apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: prod
spec:
  type: ExternalName
  externalName: prod.database.example.com
---
apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: dev
spec:
  type: ExternalName
  externalName: dev.database.example.com

Now all the service-owners have to do is point their code to database.svc.cluster.local and they will talk to the right database every time.

database.svc.cluster.local = CNAME test.database.example.com

It’s worth noting since the pod requesting the service will get the CNAME record back, the pod also has to be configured to resolve whatever domain the CNAME returns. If it’s a private record, you might want to configure revolvers in CoreDNS.

How Virtual IPs work

Most services (sorry, ExternalName) use a virtual IP to forward requests to the proper pods. It’s the responsibility of kube-proxy to provide this virtual IP and make sure the traffic is actually forwarded. kube-proxy runs on every node so the steps below happen per worker node.

How this works depends on the mode of kube-proxy is given with -proxy-mode flag:

ℹ️ For all of these kube-proxy will monitor the control plane for new/updated/removed Service and Endpoint resources and run the following steps

userspace (legacy)
- For each Service it opens a random proxy port on the node
- This proxy port is then told to iptables to capture traffic to this particular virtual IP and port to redirect to the Pod IPs in the Endpoint resource
- The backend pod is chosen based on the SessionAffinity of the service
- The traffic here is actually proxied via kube-proxy itself
iptables
- For each Service it creates iptables rules which redirect traffic to the virtual IP and service port
- These redirect rules point to sets of pod IPs based on what is in the endpoint resource
- The backend pod is chosen at random by default
- The traffic is all handled within the Linux netfilter so all within the kernelspace, kube-proxy doesn’t handle the traffic it just sets up rules to process it
IPVS
- Whilst iptables was designed to be a firewall and kube-proxy just uses it to redirect network traffic, IPVS was made for load balancing
- At a high level this works similar to iptables, cluster IP and port → set of pod IPs
- IPVS is important when working at scale, if you have >1000 services (or 10,000 pods) in your cluster IPVS is the performant choice
kernelspace (windows)

Bringing it all together

Let’s create an example LoadBalancer service and see what components come together to get traffic from a client to your pods. We’ll show how traffic from outside, via the load balancer, gets to your pods and how another pod in the cluster can also use this service without going via the load balancer.

In this example let’s say we’re running on a Kubernetes cluster in AWS with the AWS cloud controller manager properly configured, we have pods running with label app=alpha which listens on port 8080 for traffic and kube-proxy is in the default iptables mode.

Service YAML

apiVersion: v1
kind: Service
metadata:
# This doesn't affect routing but it's best practice to label components together
  labels:
    app: alpha
  name: example
spec:
  ports:
  - name: main
    port: 8081
    protocol: TCP
# Target port is the port your pod has open
    targetPort: 8080
# This is what tells the service which pods to route to
  selector:
    app: alpha
  type: LoadBalancer

Lets create the service and have a look:

$ kubectl apply -f service.yaml
service/example created

$ kubectl get service example
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
example   LoadBalancer   10.43.205.153   192.168.0.147   8081:32255/TCP   4s

I now have:

An Endpoints resource corresponding to ready pods matching my label selector
- kubectl get endpoints example
A cluster IP setup for my service 10.43.205.153
A DNS address for the service configured
- (from inside a Pod)
- dig example.default.svc.cluster.local
A external (to the cluster) IP which maps to my load balancer 192.168.0.147
A NodePort setup 32255 so all my workers now have this port open
- netstat -tulnp | grep 32255
iptables rules setup
- sudo iptables -L | grep example
- sudo iptables -L | grep 10.43.205.153

Lets see how this request goes through to our pod from outside the cluster and a pod within.

From outside the cluster

The client makes a request to the load balancer to try and talk to the application, for how this is configured see my post on ingress into a cluster
The load balancer selects one of the configured listeners, this is usually either random or round robin but depends on your load balancer and configuration
The node has port 32255 open, requests here are configured in iptables to forward the request to the virtual IP of the service. These rules were configured by kube-proxy
iptables then resolves the virtual/service IP into a Pod IP address, one is chosen by random and the request heads there. Now the traffic is inside the cluster the CNI controls the network movements between nodes (see here for more info on what the CNI sets up)
Once the request has reached the node with the pod on it, the CNI and iptables will forward it to the Pod as expected

From a pod within the cluster

The requesting pod will have been configured with the service DNS address example.default.svc.cluster.local. As it makes this request from within the cluster CoreDNS running will be the DNS resolver. This request will be resolved to the Service (virtual) IP address: 10.43.205.153
Making a request to this address will hit the rules in iptables, configured by kube-proxy, this resolves the service IP into a Pod IP by randomly selecting a pod
The rest of the request continues as in the external example.

The case for time-tracking

dom@domgoodwin.dev (Dom Goodwin) — Sat, 05 Feb 2022 15:06:09 +0000

Time tracking has a bad reputation, it invokes images of managers watching over your shoulder or pay being docked as you punched in minutes before your shift ended. Due to these, and many more, if you suggest to someone to start tracking their time they’re likely to look at you like you’ve gone mad.

I think time tracking, for your own benefit, can be a great way to be introspective about how you work and allow you a greater understanding of managing distractions, context switching and when you’re feeling productive.

What is time tracking

The basic gist is, you start a timer when you are working on something. Whilst you could track time per ticket or some other fine grained metric, I would strongly recommend tracking categories of activities.

Ones I track are:

Development - directly coding, usually “when I’m in my IDE”
Live support - picking up alerts that have fired or investigating issues/incidents
Messaging - slack messaging people
Meetings - time spent in any meetings
- I found it in the past useful to tag these “ceremonies”, “1-1s”, “other”
Break - Going to grab a coffee or lunch
- This one could be tracked by just not running a timer, but I find ending the Break timer when I return is a useful trigger to start working
Collaboration - Working with someone else to either plan, pair or debug

These activities can also be expanded with tags if you want to know more detail, things like whether the activity is for client a or b can be useful if that fits your environment.

I’ve also found adding activities when you want to focus on a particular area, say “Documentation” or “Reading tech books” can help you focus energy towards a goal.

A key thing with the tracking is not being too detailed, it’s not useful to know ticket-12 took twice as long as ticket-14 in the scheme of things (even if Toggl do like to push jira integration). You’re not billing people for this time, it’s just about gaining understanding.

How to time track

This might seem like an obvious area “set timer to start when you work on something” but there’s some nuance and tips I can share.

If at any point you stop working on your current task for distractions, stop said timer (and set a new one if applicable)
- A common thing might be you need to respond to someones message, ideally you try to batch these where possible and the people messaging you don’t just say hi, so maybe after 30minutes, stop the timer, start the messaging timer and respond to people
Before you begin any activity, start the timer
- I find this is a useful way of getting in the mindset for that task too, it forces you to mentally think “what am I working on right now” instead of just falling into work
You can always update past timers if you forget too, don’t try to be too accurate, to the nearest 5 minutes should show you the gist of it.
“Doesn’t this add to the context switching load?” Yes I believe it does, but that isn’t a problem. It should be harder for you to switch what task you’re working on to avoid (where possible) context switching and having to recognize you’re doing it, stop and start a timer, adds to the overhead, it’s way too easy to down tools on work to respond to slack.

What time tracking is not

It’s important to call out what personal time tracking is not, and avoid some of the pitfalls.

Time tracking should be for your own benefit, and you should be the only one who sees the data. You’re not doing this to prove to your manager at year-end reviews you keep working even when you sneeze or to compare to your team. On the other side, this shouldn’t be something a manager imposes on you and wants to see progress reports.
Related to above, it’s important to be honest when time tracking. Everyone has bad days for productivity and just leaving “development” timer on won’t help you understand anything. This timer is just meant to be data on how you work, no opinions or judgement.

Benefits of time tracking

I’ve personally found tracking my time, even for short 1 month stints, a great way to introspect on your working habits.

A good example is at the end of the day, when you might be feeling like it was (or wasn’t) a productive day. Being able to look back and go “oh, those 4 meetings 30minutes apart basically translated to 4hrs of meetings for my productivity”. I’ve commonly found days when I’m feeling unproductive, I can look at my time tracked for the day and understand why.

Another key aspect is seeing the effect interrupts have on you, everyone knows the bane of context switching at this point, but being able to see the switch in activity and then the lull between starting anew is clear.

I’ve also found having to think about starting a task, having a timer to start when I go into development mode helps mentally adjust to the task at hand.

Tools and use and how to set them up

For actually tracking the time, I use: toggl track. Honestly, I don’t particularly like it but I haven’t found an alternative that does everything I need and don’t need to install.

I’ve previous used Timery, which makes it easier to change timer and highly recommend it to never interact with toggl. I would highly recommend this when you can install applications on your own machine and for iOS if you have it.

I have a pinned tab open in my browser with toggl, it goes grey when there isn’t one active which I find useful
- Timery will sit in the top bar of my Mac which makes using it easier
I then setup a widget on my phone home screen. This helps me spot any timers I’ve left running.

Within toggl, I have the activities I listed above as “Projects”. I don’t use any tags currently but I’ve found it useful in the past when working across teams or clients.

A good rule is a project should be something important to you but broad enough you don’t need to think or decide which one to use when starting a timer. It should be obvious what the work you’re doing is.
As an example for tags, I wanted to see which client’s live issues were taking the most time in my day.

To round up

Time tracking can be a useful tool to gain insight into your working habits and productivity. I’d recommend only doing it when you’re gaining something from it; in the past I’ve used it for 1-3month stints when feeling unproductive or joining a new company. If you’re doing it and finding the data not useful.

Kubernetes Networking #1

dom@domgoodwin.dev (Dom Goodwin) — Mon, 10 Jan 2022 12:06:09 +0000

Kubernetes networking can be a bit of a rabbit hole to debug and discover what your pods are actually doing. In this post we’ll explore at a high level how a request goes from 1 pod to another and which components control each.

Components

The components we’re going to look at today are:

kube-dns/coredns resolves domain names for services (and pods) into an IP address
- ℹ️ kube-dns was replaced by coredns but the in cluster service is still called kube-dns
- This is the resolver for any DNS requests inside your pods, if you wanted to resolve internal service domains or to a private DNS resolver, you could configure this to forward those requests.
kube-proxy resolves network requests for the IP address of a service to the IP addresses of pods which match that service. This has different modes:
- iptables (default) - To see your iptables rules: iptables -L
- ipvs - To see your ipvs rules ipvsadm -Ln
Your chosen CNI, there’s a range of these, all which work differently. Generally a CNI will configure any networking around a Pod IP, routing to and from pods to:
- Ensure pod network requests are routed properly to other pods
- Ensure pod network requests can reach outside the cluster

An easy way of thinking about it can be, resolving between a service and healthy pods is kube-dns and kube-proxy but then that request will need to go through routing setup by the CNI. For cloud environments, your CNI will usually handle (or provide ways to setup) things like cross-subnet traffic as well.

To follow the resolution between components, say we wanted to go to a service called myapp which sits in namespace namespace001 which had 2 healthy pods matching the service selector. The resolution would be:

CoreDNS: (service FQDN)myapp.namespace001.svc.cluster.local → 172.16.10.10 (service cluster IP)
iptables: 172.16.10.10 → 10.10.10.10 (a pod IP, selected at random from healthy pods)
CNI: Will route traffic for 10.10.10.10 from the requester node, to the target node

Example journey of a request

As an example, let’s explore in an example Kubernetes cluster how a pod will send a network request to another pod. For this example:

The pod will try to talk to a pod on another node then itself
The 2 nodes will be in seperate subnets
- Routing between subnets is presumed to be setup and fine and we’ll abstract that to a router in the diagram
The CNI will be Calico, but we won’t be looking too much into its internals
pod-0 will send a request to pod-3 through a service
We’ll assume Kubernetes and Calico are setup with defaults:
- iptables mode for kube-proxy
- vxlan encapsulation for cross subnet traffic

1 pod-0 requests the service’s Cluster IP from kube-dns
- kube-dns has dynamic config which updates mapping the service name to a Service IP address lets say 172.16.10.10
  - `kubectl exec command for coredns
- *technically this whole request to kube-dns goes via the same iptables→routing tables path since kube-dns runs in pods on the cluster. The only key difference is the DNS resolver is an IP address so it doesn’t ask kube-dns to define kube-dns. But trying to show aswell would be madness.
2 With the cluster IP (172.16.10.10) the request hits iptables
- iptables selects from a random one of the pod IP addresses for this particular service IP
  - iptables -L or iptables-save
- These rules are all loaded in here and dynamically updated by `kube-proxy If a pod starts failing Readiness checks for instance, it gets removed from this list.
3 The Pod IP is now resolved in the nodes routing tables sudo route -n or ip route list
- These routing table rules will be configured based on the network environment the node sits in. Calico, running locally, will also configure rules for pod networking.
- Based on the Pod IP, it’ll be resolved through multiple sets of rules:
  - This node will have a range of Pod IPs which resolve to the local node itself
  - This node will also have an IP range for nodes inside the same subnet - these could avoid encapsulation for performance gain
  - This node also has range(s) for other subnets, if it’s not local to the node or subnet, it then gets routed to the subnet routing table. For these rules, Calico might instead route traffic to itself to be encapsulated before crossing the network boundary.
  - There also might be a range for external traffic not local to the network, but if this was a private subnet this is most likely behind a NAT gateway inside the local network
- This particular request is destined for another subnet. The routing table has directed the request to the Calico pod running locally on this node to handle routing this.
4 Calico will now encapsulate the network request to allow it to pass subnet boundaries calicoctl node status
- By encapsulating the request’s original networking information is wrapped up as data and new network info is added
- Calico will send this as a vxlan or ipinip packet to the required node in another subnet
- The encapsulated packet (shown as an orange line) will:
  - Go back through the route table on the node itself
  - To the local subnet router, which will resolve which subnet it needs to go into
  - Hit the destination subnet routing table which can direct it to the node
  - To the other node itself
5 Once the encapsulted packet arrived at the other node, it’ll go through iptables rules to see if it’s allowed traffic.
- This is why you might need to add the encapsulated packets into your firewall rules to be allowed in.
6 The routing table on this node will now:
- Send the encapsulated packet to the local Calico to be unencapsulated
- The original (unencapsulated) network request will resolve again in iptables and routing tables rules to determine where it should go
  - This could also cause the packet to have to be routed again since the destination has moved
- The request is directed to the pod running locally

There are a lot of things missed off here:

Technically iptables is interacted with before, after (and sometimes during) a routing decision
How the pod sits within it’s own network namespace with a virtual ethernet interface for traffic coming to the pod
How Calico uses Felix to program it’s routes and ACLs and BIRD for distributing the routes to BGP peers on the network
Alternate CNIs
How the whole kube-proxy aspects could be stipped out for an eBPF approach (Calico’s eBPF dataplane or Cilium and it’s kube proxy replacement)

Blog² - Blog about blogs

dom@domgoodwin.dev (Dom Goodwin) — Mon, 03 Jan 2022 12:06:09 +0000

Here’s a list I (hopefully) update around tech blogs I enjoy

Kubernetes Debugging

dom@domgoodwin.dev (Dom Goodwin) — Mon, 20 Apr 2020 14:57:09 +0100

This is a workflow to debug common Kubernetes problems when deploying applications.

Click on the image to see it in further detail

Shellscript Notes

dom@domgoodwin.dev (Dom Goodwin) — Thu, 20 Feb 2020 15:25:00 +0100

Useful shellscript snippets

Commonly used

# Check if variable contains mystring
if [[ $var == *mystring*  ]]; then # The double closed brackets are essential for wildcard match
    echo "it does";
fi

# Split on ":" and get 1st
echo "docker-image:0.1" | cut -d ":" -f1 # returns docker-image

# Find and replace all via pipe
echo "Hello I'm Dom" | sed 's/Dom/<void>/g' # "/" could be any char


# Redirect stdout and stderr to the void
./run_script.sh >/dev/null 2>&1

Basics

# Read top lines of file
head -n 3 myfile

# Read bottom lines of file
tail -n 3 myfile

# Watch all updates to file
tail -f myfile

# Read file in window (not cat)
less myfile

# Make all directories if not exist
mkdir -p exist/not/dir

# Visualise dir structure
tree -L 2

# Remove env variable
unset myvar

# Alias command to short one
alias lc="ls -laht" # unalias to remove

# Piping output
echo "overwrite" > file # Same as 1>
echo "append" >> file

System

# See disk space usage (human readable)
df -h

# See file usage for dirs
du (--max-depth=2)

# Show running processes, a: show other user, u: show username, x: show non terminal
ps -aux

# Show usage
top

Tricks

# Use command output as if it's file
diff <(grep string file1) <(grep string file2)

# Repeat last 3 command
!!

# Repeat last arg of the last command
!$

# Take all args from previous and use
!:1-$

IO

# Secure input
read -s -p "Password: " password

# Default value
read -p "Address [http://127.0.0.1]: " ADDR
default="http://127.0.0.1"
ADDR=${ADDR:-$default}

jq

# Get value from object within array within object
jq -rec ".parent.children | .[] | .name" # raw:no quotes, e:null exit0, compact

# Get object attr and value and format "Key=$attr,Value=$val"
jq -r '.data | keys[] as $k | "Key=\($k),Value=\(.[$k])"'

# Use output in for loop
for object in $(echo "${JSON}" | jq -rec ".entries[] | .whole | @base64"); do
  echo "This is the value: $(echo ${val} | base64 -d | jq .attribute)"
done

# Args
curl localhost:8500/v1/catalog/nodes | \
    jq \
        --arg ip $(./get_host.sh) \
        '.[] | select(.Address==$ip)'
        
# Output mulitple entries in list
jq -rec '[.importItems[].itemId]'

# Create object
jq -rec '{} +{ Items:[.importItems[].itemId]}'

Behaviour

# Exit script if commands return nonzero
set -e

# Output command as they run (debug)
set -x

Podcasts reccomendations - 07/2019

dom@domgoodwin.dev (Dom Goodwin) — Mon, 22 Jul 2019 11:28:19 +0100

These are my currently podcast reccomendations for Q2 2019

The Teacher’s Pet

A great true crime podcast exploring a 36 year old cold case around a woman who went missing and who’s husband brought his 17 year old, school girl, girlfriend into the house within 2 days.

Episodes to get started

Ep. 1: Bayview

The Weeds

Podcast from Vox where they do a deep dive into a topic for ~1hr. If you pick an episode with an good topic usually very informative and interesting listen.

Episodes to get started

Americas two housing crises

Into the weeds with Elizabeth Warren

Gangster Capitalism

Series 1 of the podcast covers the admissions scandal with some truly horrific stories of entitled parents. The podcast really going into the inner works of the scams being done and the unprecedented scale of the system in place.

Episodes to get started

S1 [1] The Side Door

The Shrink Next Door

A deeply interesting and weird story around a therapist who abused his position with his patients. Focuses mainly on this one individual who got taken advantage of to a ridiculous level, leading to him being ostracised from his family and friends.

Episodes to get started

Welcome To The Neighborhood | 1

Dissect

Season 4 of Dissect covers Tyler the Creators album, Flower Boy. An amazing and indepth look into this brilliant album with some bonus episodes covering some of his over work including his new album, Igor.

Episodes to get started

S4E1 - Tyler the Creator: Flower Boy

Dissecting Igor with Anthony Fantano

Podcast reccomendations - 02/2019

dom@domgoodwin.dev (Dom Goodwin) — Tue, 12 Feb 2019 14:31:03 +0000

These are my currently podcast reccomendations for Q1 2019

The Butterfly Effect

Jon Ronson’s Audible exclusive (now available everywhere) where he explores the affect PornHub had on the porn industy. Very insighful and interesting, each episodes covering a broad area such as the impact on kids in school. Only 7 (+1 bonus) episodes but still covers a lot.

Episodes to get started

Ep. 1: A Nondescript Building in Montreal

The Last Days of August

Following on from The Butterfly Effect, released this year on Audible (public release in April), Jon explores the death of a famous porn star and the mystery surrounding her death and her husband.

Episodes to get started

Ep. 1

Bear Brook

True Crime stories focussed around two barrels of bodies found in the woods. The podcast covers how crime investigations methodologies were changed forever by the techniques used.

Episodes to get started

Ep. 1: Hide and Seek

This American Life

A classic podcast but some recent episodes are worth a mention, first covers

Episodes to get started

#661: But that’s what happened

#657: The Runaways

#486: Valentines Day (Act 2 in particular)

Cortex

Two youtube/podcast company owners discuss productivity, goal planning, efficiency and technology

Episodes to get started

78: State of the Apps 2019

72: Adulting Complete

79: 2019 Yearly Themes

About

dom@domgoodwin.dev (Dom Goodwin) — Sun, 27 May 2018 17:12:07 +0100

Dom Goodwin Backend Engineer @ Monzo

Setting up a Hugo Blog

dom@domgoodwin.dev (Dom Goodwin) — Fri, 06 Apr 2018 11:03:53 +0100

Hugo is a Go based, static site generator. It has some impressive benchmarking figures, a huge catalog of impressive themes and means I don’t have to write HTML.

I’m setting this up with the follow configuratons

Ubuntu 16.04 LTS
Hugo 0.32
Go 1.9.4

Installing Hugo

Ubuntu/Debian:
sudo apt-get install hugo

Check it’s installed correctly:
hugo version

Creating your site

Basic setup

To generate your base site:
hugo new site my-blog

This will create a folder, my-blog, in your current directory

The folder structure will be:

+--archetypes/     Templates for content files  
+--content/        Where content for your site goes  
+--data/           Config files for generating your site  
+--layouts/        Html templates for content views  
+--static/         All static content for site goes here  
config.toml        General site properties like: theme, title

Setup your site as a git repo:
git init

Using a theme

Pick a theme you want to use from here

As themes are stored as git repos, the best way to use one is a submodule
git submodule add {THEME-REPO} themes/{THEME-NAME}

Once the submodule has been added, add this line to your config.toml:
theme = "{THEME-NAME}"

To see what your theme looks like, run the Hugo server:
hugo server

Your empty site should be availbale at:
localhost:1313

Adding content

To create a new post for your site:
hugo new post/hello-world.md

By default (because of the archetypes/default.md) your post file will look like this:

title: "Hello World"  
date: 2018-04-06T11:03:53+01:00  
draft: true

To see your page, you need to run the server with drafts enabled:
hugo server -D

Overriding themes

You might want to change how a theme renders something. To do this, it’s easy. Just copy the files you want to change from:
themes/{THEME-NAME}/layouts/ to: layouts/

Then the files in the root layouts folder will override your theme ones

Podcast reccomendations - 04/2018

dom@domgoodwin.dev (Dom Goodwin) — Wed, 04 Apr 2018 22:03:36 +0100

These are my currently podcast reccomendations as of April 2018

Reply All

Every week, you’re presented with an intersting tale relating to the internet. From online bounty hunters to crowdsourcing medical advice, this podcast’s range of stories are always interesting and worth a listen.

Episodes to get started

Long Distance : #102 / #103

The Secret Life of Alex Goldman : #96

Boy in Photo : #79

This American Life

This American Life is one of the most popular podcasts of all time. With over twenty years worth of archives, the amount of amazing episodes are endless.

Episodes to get started

Human Error in Voltile Situations : #634

20 Acts in 60 Minutes : #241

Five Women : #640

My Favourite Murder

Every week, the hosts tell some of their favourite, most interesting true crime, murder stories.

Episodes to get started

My Firstest Murder : #1

Decompressions : #102

Sawbones

A look into the history of how illnesses and medcial problems were solved throughout time. Makes you wonder if we’re as wrong now as our ancestors were.

Episodes to get started

The Women Who Gave Birth to Rabbits

Opioid Addiction

No Such Thing As A Fish

The researches behind the facts for QI talk about their favourites facts of the week.

Episodes to get started

Really any, they’re all great

Page 94: The Private Eye Podcast

A more detailed look into some stories within Private Eye.

Episodes to get started

Hospital and Hacks

Credit and Christmas

Disect

Long form music analysis, each season focussing on an album.

Episodes to get started

S2E1 - Kanye West: The Elephant in the Room

S1E9 - Alright by Kendrick Lamar

Hello Internet

Two very inteligent, insightful and interesting people discuss topics of note; always thought provoking.

Episodes to get started

Being Wrong on The Internet, #1

Mr Chompers, #93

Freakonomics

Two economist explore the hidden side of everything, from pizza stores to chuck e cheeses.

Episodes to get started

What Are you Waiting For?

How Big is My Penis? (And Other Things We Ask Google)

My Dad Wrote A Porno

I think the title of this one really says it all.

Episodes to get started

S1E1 - ‘The Job Interview;’

Mortified

Adults tell the embarrassing stories of things they did as kids.

Episodes to get started

Titanic Ruined My Love Life, #88

How to Become a Catfish, #105

Notable mentions

S-Town, Serial, Crimetown

Books

dom@domgoodwin.dev (Dom Goodwin) — Mon, 01 Jan 0001 00:00:00 +0000

Currently reading

Read

To read

Solar

dom@domgoodwin.dev (Dom Goodwin) — Mon, 01 Jan 0001 00:00:00 +0000

Solar Generation

13 405W panels
5kW inverter
Days with 0 data are when the monitoring system was down

Dom Goodwin

Don’t just say ‘hello’ to me

Reducing our NAT Gateway cost with private networking between AWS and GCP

Kubernetes Networking #3: Ingress

🤷 Why not just LoadBalancer?

🛠️ So what does an ingress controller actually do?

🔒 What about SSL?

🎛️ Load balancer/Reverse proxy controls

🤹‍♂️ Can I have multiple ingress-controllers?

📮 What about hostnames?

external-dns

🍵 Bringing it all together

🦦 Other controllers are available

Kubernetes Networking #2: Services

Selectors

Service Types

ClusterIP

NodePort

LoadBalancer

ExternalName

How Virtual IPs work

Bringing it all together

From outside the cluster

From a pod within the cluster

The case for time-tracking

What is time tracking

How to time track

What time tracking is not

Benefits of time tracking

Tools and use and how to set them up

To round up

Kubernetes Networking #1

Components

Example journey of a request

There are a lot of things missed off here:

Related docs:

Blog² - Blog about blogs

Kubernetes Debugging

Shellscript Notes

Commonly used

Basics

System

Tricks

IO

jq

Behaviour

Podcasts reccomendations - 07/2019

The Teacher’s Pet

Episodes to get started

The Weeds

Episodes to get started

Gangster Capitalism

Episodes to get started

The Shrink Next Door

Episodes to get started

Dissect

Episodes to get started

Podcast reccomendations - 02/2019

The Butterfly Effect

Episodes to get started

The Last Days of August

Episodes to get started

Bear Brook

Episodes to get started

This American Life

Episodes to get started

Cortex

Episodes to get started

About

Links

Setting up a Hugo Blog

Installing Hugo

Creating your site

Basic setup

Using a theme

Adding content

Overriding themes

Podcast reccomendations - 04/2018

Reply All

Episodes to get started