The British love to drink. But how many have you have ever stopped to wonder, how many grams of carbohydrates are in a pint of beer?

What about in this meal?

And what about this salad, that is usually listed on a menu as the low carb option?

One answer you might give reading this is who cares?

People only care about the amount of calories they are eating if they are trying to gain or lose weight. Maybe you care about carbs a little bit if you are following the Keto diet (but even then, a lot of “keto” meals do have carbs in them).

Well, it turns out there is at least 8 million people globally who really care about the answer to these questions, and I’m one of them.

In 2020, I was diagnosed with Type 1 diabetes. Likely you have heard of diabetes, but you might not be too familiar with type 1, which is a rarer form of it. Before diagnosis I wasn’t too familiar either, so here’s a brief primer.

Type 1 in 100 words

Type 1 diabetes is an autoimmune condition where the pancreas produces little to no insulin, a hormone essential for converting carbohydrates into energy. This means Insulin has to be injected to supplement the fact that you don’t produce it.  How much do you inject? It depends on what you are eating and a bunch of other variables that are so discreet it sometimes feels like it’s random.

The Insulin pen I use before eating. On the right you can see a configurable dial to determine the amount of units to inject.

Type 1 is not caused by lifestyle, and it has no cure. In practise this means Type 1 diabetics have to monitor their blood sugar at all times. If it is too high for too long, then it could lead to long term damage and a shortened life span. If it is too low for even a short period, then it can be fatal. Sometimes if it gets too low I am effectively paralysed and am unable to eat or drink safely and I will need support. Thankfully, for me at least, this is rare. However, it's a condition you very much need to respect and never take your eye off the ball with as one mistake really could be fatal.

With type 1, simple things are taxing. Whenever I see food or a drink like those above, I’m doing mental gymnastic to try and figure out how much insulin I need to inject and what impact that will have on my day. For example, If I want to go for a brief walk, I need to think about how much active insulin I have in my system as my blood glucose levels tend to drop quite aggressively with even moderate exercise.

I hope the start of the blog didn’t seem too “gloomy”, I have tried to be as factual as possible. The reality is I have a pretty optimistic outlook. From research it seems there has never been a better time in history to be a type 1 diabetic. I hope that every year that goes by I get to say that and mean it more.

Even in the short time since diagnosis,  technology has come a long way for health monitoring and I can now monitor my blood glucose wearing a device on my arm. Here is a picture of me wearing it and yes, I am available for modeling work.

I have to change this every 2 weeks, and it is essentially a little pin that goes into my arm. Here’s a picture of it before application:

The device (Called a Libre) takes a reading of my blood when I tap my phone against the device. If it goes below the red line in the image below, I need to eat some carbs. If it goes outside of the green range, I may want to take some action, including taking some insulin.

It gets a little more complicated than this though.  As mentioned, doing exercise, gaining weight or being sick changes insulin sensitivity and so management becomes a moving target. Its therefore essential that I have a dynamic approach towards management. Furthermore, sometimes this happens.

During this period, I am not able to take my blood glucose levels using the device and I also would not receive alerts, if my blood sugar was to reach dangerous levels. This error had the habit of showing up at the most stressful times. 

Especially in the first few months of diagnosis, I was really struggling to control my blood glucose and was completely dependent on the monitor. Even today I have periods when my blood goes low and I can’t figure out why. 

If visibility was to drop to 0 like this at work, we’d declare an incident and we would not close it until visibility was restored. Furthermore, my blood glucose going low is the first signal that something is going wrong and some action should be taken.

 I wonder if I can apply the same thinking we apply to incidents, to my blood glucose?
A Brief Detour into Incident Management

Although being diabetic does feel all-consuming at times, I do work full time as an Engineering Manager at Cloudflare.

At Cloudflare, If anyone in the company (or an automated system) detects that something is not working as it should then we raise an incident. It's not quite a big red button that we press, but it is not far off. We have developed an internal bot that can be used to raise incidents, manage the process and generate incident reports at the end.

To detect and raise incidents, we use the familiar suspects of Prometheus, Grafana, Alertmanager and Pagerduty, as well as some of our own custom tools. If you have not used these tools before, here’s a one sentence summary of their role.

• Prometheus is an open-source tool used for event-monitoring and alerting. It’s incredibly powerful and a great option for storing time-series data in.

• Grafana works really well with prometheus and allows you to build beautiful dashboards over the top of the data.

• Alert Manager allows you to configure alerts for specific queries to trigger pages.

• PagerDuty is the only application on my phone that gives me dread when I see a push notification for it.

Every Engineering Manager and some senior contributors receive training on what we call being an incident commander. It’s not quite as glamorous as this image generated by DALLE 3 but I do know some engineers who have widescreen monitors as big as this now.

As an incident commander, you are empowered and encouraged to pull anyone in from anywhere in the company to resolve the incident as quickly as possible. By resolve, I mean remove customer impact which means you must identify, triage, fix/put a short term band aid on and identify and record long term actions. We then do a review of every incident to ensure we learn from it and take the necessary steps to ensure it does not happen again.

Successful incident management is all about collaboration and working together towards a common goal, which is usually restoring established behaviour.This may mean engaging with a subject matter expert or escalating it. In an ideal world, you engage as few people as possible to resolve the incident, as being paged can be pretty disruptive. However, it’s much less disruptive than ignoring the signals and trying to repair something that has now completely broken.

I wanted to apply this same thinking to managing my blood glucose. If something happens which causes my blood glucose to trend downwards, I wanted to be notified so that I can take an action before any more major issue appears. However, if things do take a turn for the worst or I’m unable to self-resolve, I wanted to feel good about the fact that someone else would get notified if I am unable to help myself and hopefully prevent it from escalating from a P3 to a P0 incident.

Type 1 Incident Management

I really wanted to get to a place where i felt confident that if my device was taking no readings or I was critically low, myself or someone would be notified regardless.

The device I use is closed source, and does not give an easy way for you to build on top of it. It does not expose any APIs, and certainly has no SDK. I spent some time messing around trying to get the data off of it but did not find an obvious solution. I started to explore other solutions. After some research I found this device called a Miao Miao.

It slots over the top of the device on my arm and effectively “scans” the libre every 2 minutes and sends the result to another app that is called tomato. The really nice thing I found about this that even when the main device would show me the error, this one would continue to publish data. After playing around it seemed that the main app didn’t like sending data that it felt was anomalous due to a sudden change in blood glucose level. However, I personally would like to know about this, or atleast would like to make the decision myself of whether it is useful rather than it being made for me.

Natively, the device also includes a really clever “hack” that effectively published your blood glucose to Google calendar as events every 5 minutes. By doing this, you could use Apple’s native complications to see your blood sugar on your Apple Watch. I found this really exciting and it immediately got my brain whirring as this meant their must be a way to submit blood glucose readings to a third party API, I just need to figure out how to send it.

Engineering a Solution

Whilst looking through the app, I saw in the settings that it had a data sync option and you could enter a URL, intended to be used for an open source tool called nightScout.

Instead of entering a Nightscout URL, I pointed it at a webserver I had running. I entered the URL and hit submit, and I saw these log lines in the Gateway. I now knew what endpoint the device was trying to submit data to but not what that data looked like.

To solve this, I wrote a simple echo server using Go that matches the route that we saw previously.  To deploy it I’m using encore.dev which allows me to deploy and run my monitoring system for free. The annotation at the top is instructions for encore on where my httpHandler will run.

An echo server is a server that sends back the same data it receives from a client. In other words, whatever you send to an echo server, it "echoes" or reflects back to you. The concept is simple but they can be really useful for network troubleshooting, application behavior testing, or something like this.

Echo Servers are a useful tool in any language, but they are exceptionally easy to write in Go. This one is barely 3 lines long and does exactly what we need.

I repeated the same exercise for some of the other requests the device made and came across one that gave this response. This is perfect.

• SGV is my blood glucose. In other countries they use a different measurement than we use in the UK so 73 here needs to be divided by 18 to give the scale I use in the rest of this blog.

• It gives a date time.

• It gives a trend direction.

• This call was being made every 2 minutes.

Now we have an endpoint we can submit readings too, I made a gauge and set my blood sugar everytime I receive one.

A gauge is a metric that is exposed that represents a single numerical value that can go up or down.

It is used for measuring values that can fluctuate, such as the current memory usage, number of concurrent requests, temperature, or, in this instance, blood glucose.

I also “best efforts” store the data in Postgres (which is the insertReading function). This is one of the few use cases I have come across when the DB persist is not too important to me, and I don’t want to fail the rest of my logic because of its failure. I therefore log it to acknowledge it and move on.

I can now have my blood sugar being submitted to an endpoint every 2 minutes and setting a metric.

Now I have this gauge being updated every couple of minutes, I can now make a Grafana dashboard which shows my blood glucose in real time. The Y axis shows my blood glucose level. I want to keep this value between 4 and 9 ideally.

As you can see I can view trends over time and share it with anyone. I could even setup a monitor at home for this dashboard so I can glance at it.

So now I can share my blood glucose data with anyone, but its kind of useless in isolation. The same as monitoring any complex system, context is important. For example, if:

• I have just eaten

• there is a spike

• … but I have already had insulin

its nothing to be immediately worried about as Insulin takes a bit of time to work. I therefore wanted to build a quick way to add this context.

Grafana supports annotations, but I didn’t want to login to annotate and wanted it to be less of a burden. I therefore build a chatbot using telegram that enabled me to add annotations. Telegram can be configured to send a webhook every time a bot receives a message. Here’s what that looks like.

The webhook receives a request, I validate the message, unmarshall it and write back to telegram to let me know if there was any issues processing it.

After the validation I attempt to parse ints and then call my annotation service, before writing back to Telegram and also returning an ok.

My annotation service assigns tags to the message based on its content, and sends these to Grafana.

In Grafana, I can assign colours to these different tags and setup annotation queries.

We now have all the pieces for it to work! Here’s a gif of me chatting to the bot and the annotations showing up on Grafana

I now have a way to add context to my graph which is really powerful for both myself, and other people viewing it to understand what is going on with my blood glucose. If I do have a low blood sugar event, then there is lots of context around it which is really valuable to understand what action to take.

What about alerting?

So we now have great visibility, but no alerts. Guidance given to me by nurses is that when my blood glucose goes below 4, I should start treating for it. I therefore wrote a tiny cron that checks my readings every 5 minutes and if a reading is less than a pre-defined limit, it triggers an incident.

Encore makes this very easy, and you define cron jobs in code like this.

I also added this block of code to my telegram bot to allow me to manually trigger an incident if for any reason I needed to.

This is what that looks like.

But what actually happens when I trigger an incident?

When I trigger an incident, it calls my incident microservice which opens an incident incident.io. I picked incident.io as it is the closest tool to Cloudflare’s in-house incident tooling and it is also written completely in Go.

The reason I wanted to use an actual incident tool is allows me to set up escalation policies, run reports to see how much time spend in “incident” state, amongst other things.

Incident.io also has the ability to run workflows. This means I can configure rules similar to what you would in Zapier, and can send notifications via text to folks who have subscribed. In this example, every time I open a workflow, it’s going to send a text message to me.

If the incident isn’t closed for 20 minutes, it will automatically escalate it to my partner or siblings, depending on who is on good terms with me at the time.

With this, I have been able to create a pretty robust incident management workflow.

As I mentioned previously, because I use incident tooling I also get reports effectively for free. This allows me to see what/how many low blood glucose incidents I have over time.

The above graph is demo data I put together for this blog as my real data was thankfully a little boring! However, you can see how powerful and useful this is. Based on the above graph, I would be able to conclude the amount of low glucose events I am having is trending up, and that would be a good signal to engage with my doctor as my current approach to treatment is clearly not working for me. Perhaps something had changed about my lifestyle and I needed to reduce my insulin regime.

Next Steps

I have only described a subset of the features of my system. Some other things it can already do or I plan to build are:

• Auto-close incidents. Right now I am manually closing them, to give me an opportunity to review the data. However, I have been to slow to do this a couple of times which has led to the incident escalating. I could easily auto-close it after 15 minutes or so of solid blood glucose.

• I mentioned previously I am storing all my blood sugar data in a database, but right now I’m not really doing anything with it. I have a couple of years worth of data now so it would be great to train an LLM on it so I can have conversations with my data. I could ask questions like “why do i always go low at 3pm?” and use other data like my google calendar to try and answer it.

• I could add way more graphs and measures. Right now my graph is simple but there are lots of measures that are used to judge your success as a diabetic, such as hba1c which i could easily add.

• More fail safes. This system is critical so the more I can to prevent failure the better.

Wrapping Up

When I first got diagnosed with Type 1, I was pretty scared. I thought it would be debilitating and it would make my life really hard. Whilst I won’t pretend it’s a breeze,  this project has really helped me understand my condition and to manage and monitor it the best way I know how; as if it was distributed system.

Being able to code genuinely feels like a superpower and has enabled me to somewhat automate management of a condition that even as late as 1920 was a death sentence.

I hope you enjoyed this tale and it inspires you to build cool stuff too. Please let me know if you do!