Matt Pallissard

Enforce terraform managed AWS tags

2022-10-19

OPA

You may or may not have heard of Open Policy Agent. Basically it’s a tool for crafting policies. It’s generalized enough that you can use it for most anything, such as ensuring that any resources created via terraform have required tags.

Rego

OPA uses Rego¹, which is in the familiy of logic programming languages. Of those, Prolog is probably the one with most recognition.

If you’ve never dabbled in logic languages I highly recommend going through one of swi prolog intros or pragprog has a decent intro to gnu prolog in seven languages in seven weeks. Unfortunately prolog has several dialects that are kinda-sorta the same, but differ enough to be a thing. Much like lisp.

Rego’s syntax is not the same as prolog. But in line with logic programming, your statements aren’t evaluated imperatively. You define the problem and let the language work out the details.

example

NOTE: The boiler plate from this example was copied straight out of the rego documentation² which is excellent overall.

You’ll notice below that there are three functions named match; a conditional selection of which to execute. So our match functions will be evaluated based on whether there is a tags object or a key named foo or bar in tags.

So the hand-wavy tl’dr;

changes walks the tree and as a side effect assigns all the resource keys in the object resource_changes to array c
match checks for the conditions as outlined above
the order this is all invoked in is dictated by the language itself

Pretty straightforward when you look at the policy itself.

#./policy/tags.rego
package terraform.tags

deny[msg] {
	match(changes[c].change.after)
	msg := sprintf("fail: %v is missing required tags.", [changes[c].address])
}

match (i) {
	not i.tags
}

match(i) {
	not i.tags.foo
}

match(i) {
	not i.tags.bar
}

changes := { c |
	some path, value
	walk(input, [path, value])
	reverse_index(path, 1) == "resource_changes"
	c = value[_]
}

reverse_index(path, idx) = value {
	value := path[count(path) - idx]
}

And the deny policy above is invoked by the cli itself.

generate a terraform plan³
run opa telling it to
- look in the policy directory for rego files
- evoke the deny policy in the terraform.tags package

~  terragrunt plan -out=./terragrunt.plan && terragrunt show -json ./terragrunt.plan > terragrunt.json
~  opa exec --decision terraform/tags/deny --bundle policy terragrunt.json 2> >(jq)
{
  "result": [
    {
      "path": "terragrunt.json",
      "result": [
        "fail: module.rds.module.db_instance.aws_db_instance.this[0] is missing required tags.",
        "fail: module.rds.module.db_parameter_group.aws_db_parameter_group.this[0] is missing required tags.",
        "fail: module.security-group.aws_security_group.default[0] is missing required tags."
      ]
    }
  ]
}

CI

Now you could wire that up in your right into your CI process using the CLI if that tickles your fancy. However, there are tools out there³ that support opa policies by default. It’s worth looking into.

Or web assembly interestingly enough. ↩︎
https://www.openpolicyagent.org/docs/latest/terraform/ ↩︎
I’m using terragrunt in this circumstance, but the process would be identical for terraform ↩︎ ↩︎

The cost of partial automation

2022-09-19

Excuses

These days businesses and teams are geared toward cranking out work as quickly as possible. From what I’ve seen, the prevailing sentiment is that time not spent adding features, or ticking check boxes is wasted time. Often this means that;

POC/spike, design, and implementation phases of a project are conflated .
Something that mostly works is labeled as good enough.
Improving internal tooling is deferred.

The most common reasons I’ve heard in order of increasing comminality;

They have never seen (and therefore don’t recognize the value) of true end-to-end automation.
Are unable to articulate the values of proper design, planning, and execution to the organization as a whole.
Thinking they do not have enough or it’s not worth their time to properly design and automate a solution.

There is an old analogy that comes in several flavors; a butcher, woodcutter, etc is busy. Their blade is dull, which slows them down. But since they are moving so slow they think that there isn’t time to stop and sharpen the blade.

As such we see people spinning their wheels. And since nobody likes spinning their wheels, we see ~~excuses~~ business justifications for wheel spinning disguised as truisms on an almost daily bases.

20% of the work gets you 80% of the way there. Premature optimization is the root of all evil. And my personal favorite; Minimal. Viable. Product. ¹

But that logic is flawed.

I’m going to call B.S. on they way most orgs use the term MVP. In essence it’s great. You want to lay out some constraints to avoid over-engineering and wasting time on uneeded features? Cool, that keeps us on task. But, more often than not, it’s used to mark tricky features that aren’t complete as out of scope.

Super common workflows labeled “good enough”

There are a lot of IaC and CI/CD automation design patterns that get labeled and marketed as good enough. What winds up happening is each portion of a design is treated as it’s own little automated unit, rather than the entire process as a whole.

For deployment changes one might look like this

PR for terraform module
PR for terraform implementation/deployment, cdktf, terragrunt, vanilla terraform
PR for helm chart
PR for chart deployment

And for changes to a core library, it might look like this.

PR for npm/pip package
PR for downstream code, of which there is often several
PR for helm chart
PR for chart deployment

And all of those little steps of opening and closing PR’s are hand-waved over since they’re “automated”.²

Napkin math cost breakdown with conservative numbers

When you start operating automation at scale³ those little bits that were hand-waved over really start to add up.

Lets pretend we have engineers with;

The average base salary in Seattle, $150,000⁴
401k match and insurance cost the employer $15,000 a year
And we’ll omit equipment, travel, reimbursments, and the costs of HR and tech support.

That’s a cost of ~$80 dollars an hour to the org.

Assuming each PR approval/merge and subsequent CI run takes 10 minutes (and there are no mistakes) you’re looking at 40 minutes for a change.

If an engineer makes two non-trivial changes a day⁵, that’s $104. $104 just fiddling with VCS/ci Web-UI’s, copy-pasting data, and running cli commands.

Lets also pretend that they take several weeks off this year.

$80 * 1.3hours * 5days * 45 weeks
23400.0

Since this is napkin math, we’ll round down. $23k in overhead a year, for each engineer. $23k a year that is adding exactly $0 in value to the organization. In fact that is money going right down the drain.

And that was conservative

Not to mention steps like this are error prone as they aren’t tightly coupled.

The PR’s have to be opened and merged in a specific order.
There is often data that has to be moved from one step to another. Manual processes.

There are going to be bugs caught by unit tests, and performance regressions, and rollbacks⁶. Things such as dns/endpoint names, services, account information, roles, secrets etc need to be shuffled around. It’s all to common to see organizations use terraform to deploy infrastructure and then manually copy-paste information into vault or a values file in helm. Or to set permissions on objects in S3 manually. Or to manually create database or IaM credentials.

If you are a DevOps engineer, work with 6 other engineers in your organization, and automate all of the ticky-tack little steps that take them 10 to 15 minutes, several times a day;

Congratulations, you improved 6 peoples day-to-day lives and it is as if the org now has an additional head count without the added cost or time investement in onboarding/ramp up time.

Not that these expressions don’t have a use. They’re just applied incorrectly. ↩︎
Which is so strange, because there are often really straightforward ways to tie that all together. ↩︎
And I’m not just talking BigCo web scale. You can have a small product and support and build tooling for a lot of teams. Scaling is not necissarily tied to your customer base. And conversely, you can have a product that is hammered hard but requires next to no infrastructure automation. Size != complexity of a given problem domain ↩︎
In 2022. ↩︎
Because if the’re only doing a bunch of little trivial things, why don’t they have the time to automate it all in the first place? ↩︎
And if it’s not 100% automated, rollbacks and hot-fixes can be awfully unforgiving. ↩︎

Code Golf: Dedup an array of unsigned integers

2022-09-15

I stumbled across some code golf the other day. A fairly typical problem. Remove duplicates from an array/stream/whatever. Rather than use a set or other typical data structure that only allows one entry, I thought it’d be fun to convert the number to base 2

uint64_t get_mask(uint64_t i) {
	return(pow(2, i));
}

Then use a bitwise or to store the result in an array.

void insert(uint64_t i, uint64_t *b) {
	*b |= get_mask(i);
}

Then checking for the existence is a bit shift and bitwise and.

uint64_t member(uint64_t i, uint64_t *b) {
	return (*b >> i) & 1;
}

Anyway, here’s the whole thing

#include <stdbool.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <math.h>

enum { LEN = 8 };

uint64_t get_mask(uint64_t i) {
	return (pow(2, i));
}

bool member(uint64_t i, uint64_t *b) {
	return (*b >> i) & 1;
}

void insert(uint64_t i, uint64_t *b) {
	*b |= get_mask(i);
}
void display(uint64_t *t, int i, int j) {
	while (i <= j)
		printf("%ld ", t[i++]);
	printf("\n");
}

int main(void) {
	uint64_t i[] = { 9, 4, 6, 32, 5, 9, 8, 2, 1, 7, 8, 12, 11, 8, 9, 25 };
	uint64_t b[LEN] = {};
	size_t h = sizeof(i) / sizeof(uint64_t);
	int j = 0, k = 0, rl = h - 1;
	display(i, 0, rl);
	do {
		if (!member(i[j], b)) {
			i[k++] = i[j];
			insert(i[j], b);
		} else
			rl--;
	} while (j++ <= h);
	display(i, 0, rl + 1);
}

Is it really automated?

2022-08-29

This is written in the context of IaC, code deployments, and cloud vendors.

What is automation?

From Wikipedia,

Automation describes a wide range of technologies that reduce human intervention in processes.

Cool. I’ve always been more of an enabler than an interventionist. Sounds like a good time, sign me up. And from the same Wikipedia page;

The main advantages of automation are:

* Increased throughput or productivity
* Improved quality
* Increased predictability
* Improved robustness (consistency), of processes or product
* Increased consistency of output
* Reduced direct human labor costs and expenses
* Reduced cycle time
* Increased accuracy
* Relieving humans of monotonously repetitive work
* Required work in development, deployment, maintenance, and operation of automated processes — often structured as “jobs”
* Increased human freedom to do other things

All good things, all good things

But lets narrow this down a bit to the things I think about when discussing automation.

Increased throughput or productivity
Relieving humans of monotonously repetitive work
Increased human freedom to do other things

So given the things that I care about, our productivity and throughput should be increased. I shouldn’t have to do the same thing over and over again, and I should be able to go off and solve new problems.

Am I alone in feeling that this is not the current state of DevOps, infrastructure, or software engineering as a whole?

Framing of the problem

Conway’s Law is an interesting lens to look at automation tooling. It states;

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Looking at a few popular IaC tools.

tool	original author	author background
puppet	Luke Kanies	Unix administrator
saltstack	Thomas Hatch	data center architect and systems administrator
terraform	Mitchel Hashimoto	software developer
helm	Dies labs	tools for enabling software developers to quickly deploy to the cloud

Puppet and Saltstack tend to be geared towards wiring up many disparate parts into a cohesive application or deployment. While Terraform is built to wire up disparate parts as well, it and helm are geared towards enabling people to leverage cloud technologies, primarily developers. Purpose and background are not exactly communication structures. But much like communication structures, they seem to be functions of how each role traditionally operates.

Despite the differences, they share common traits.

These tools accept a description of end state
From that description a delta between what exists and what must exist is created
Actions are taken to resolve said delta

But the specific focus of the tooling naturally drives design decisions. As Terraform and Helm are designed to facilitate rapid cloud deployments they are naturally built to describe individual applications or stacks. So defining my hand-wavy application foo in terraform could look something like so.

# foo.tf
terraform {
  required_providers {
    bar = {},
    baz = {}
}}

provider "bar" { }
provider "baz " { }

resource "bar_thing" "bar" {
}

resource "baz_thing" "baz" {
)]}

While most use puppet and Saltstack in a similar manner¹, you are able to break things down into their constituent parts and stitch them back together cohesively.² Which when you’re in the business of making a lot of changes en masse, is a godsend. The same application foo could look like this in salt stack.

# Yep, an empty file

Because we’d apply bar and baz to everything and use separate data to drive it, if and only if the data to drive it was defined.

{% if bar is defined %}
bar_thing:
  bar: bar.options.whatever
{% endif %}

{% if baz is defined %}
baz_thing:
  baz: baz.options.whatever
{% endif %}

We’d tell salt to walk a tree like this, merging the data basically in what basically amounts to a nested for-loop.

.
├── bar
│   ├── default.yml
│   └── foo
│       └── file.yml
└── baz
    ├── default.yml
    └── foo
        └── file.yml

And at that point you have a really nice and tidy taxonomy, if you need to change something about baz universally you have a one-stop-shop. So rather than having to copy-paste even a small amount of boiler plate and tweak the corresponding bits or feed them in as environment variables at build time, you’re relying on a series of two way merges to take a default configuration and augment or override specific parts of it.

# augments
./bar/foo/file.yml
bar:
  options:
    b: 2
./bar/default.yml
bar:
  options:
    a: 1
./baz/foo/file.yml

# overrides
baz:
  options:
    a: b
./baz/default.yml
baz:
  options:
    a: a

Which by the way, is all the way DRY.

DRY? More like damp.

Whether or not folks like to organize their taxonomy by application or it’s constituent parts, I’ve noticed that I’m not alone in the fact that lots of people like the idea of DRY³. There are even tools that set out to address this. Helmfile⁴ and Terragrunt are prime examples. Terragrunt is particularly interesting because it heavily markets itself as a DRY tool.

Terragrunt lends itself to a hierarchy, in fact, it was completely designed to cut down on duplicated terraform code. Another interesting bit is rather than traverse down or across a directory tree, it walks it upwards, so you can do something like this.

include {
  path = find_in_parent_folders()
}

And it’ll walk up the tree looking for any terragrunt.hcl folders. It’s a little annoying to have to have that include statement in every terragrunt.hcl file, and it’s not really in the spirit of DRY, but I’ll play along.

[modified for clarity]
time=2022-08-25T16:16:07-08:00 level=error msg=/home/matt/example/foo/bar/baz/example.hcl
includes /home/matt/example/terraform/foo/bar/example.hcl, 
which itself includes /home/matt/example/foo/example.hcl. 
Only one level of includes is allowed.

Just kidding, it can’t actually do that. But to be fair there is a commonly used yaml merging workaround, much like my salt example above.

inputs = merge(
  yamldecode(
    file("${get_terragrunt_dir()}/${find_in_parent_folders("foo.yml", local.defaults)}"),
  ),
  yamldecode(
    file("${get_terragrunt_dir()}/${find_in_parent_folders("bar.yml", local.defaults)}"),
  ),
)

But this is not exactly like my example above. There is a subtle difference. Remember how Terragrunt walks up the directory? This means that you have to define the yaml you’re going to merge at the bottom of the tree. So rather than being able to put in the same elements to override configuration or complements to augment it, you have to put in a big ‘ol block of boilerplate in every application definition.

Walking up has another subtle difference. Any given node can only have one parent. When walking down, you have multiple children and your are afforded more taxonomical options.

This goes back to how the tools were designed to be used. A way to lower the barrier for entry to cloud automation, not a means of centrally managing large swaths of deployments.

Admittedly, Helmfile does an alright job of this by being able to import and merge pull in some base configuration and merge input values.

bases:
  - ../../thing1.yml
  - ../thing2.yml
values:
  - ./default/prod.yml
  - ./{{ .Environment.Name }}.yml

But you’re still left with the problem of boilerplate and the additional issue of relative paths.

Event driven decisions

Due to the agentless nature of these cloud provisioning IaC tools, event driven changes are impossible. You want a something to happen and a modification happens on the fly? You can’t express that in these tools. You’re going to have to use additional tooling outside of your IaC. You might be able to describe the additional automation with your IaC, but it will still live outside the tool itself.⁵

For a contrived example; say you have a service that has metrics you can query, and based on those metrics you want to horizontally scale a database preventively. There’s a lot of requests in the queue that will eventually work their way through the system as database read operations. You simply cannot do that without a service that knows how to;

check metrics
determine that an event has happened
match that event with an action
generate a delta between the current state and the end state defined by the action
take actions to resolve the delta

This looks the similar to the list above, with one notable exception. This delta isn’t generated from a declarative configuration, it’s generated from an event. The event handler is generated from a declarative configuration, but the actual action is triggered by a listener. Which naturally has to be running if it’s going to be listening.

I’m not the only person who’s taken note of trade off’s one makes with agentless configuration. For example, CI and CD tools such as ArgoCD are always on. It’s listening for changes in a git repo to deploy applications to Kubernetes. It’s a great alternative to wiring up a bunch of weird CI jobs.

But then we separate things

But it’s yet another tool that needs to be configured. And what I’ve seen in practice is users like to separate their code by framework or tool. So it’s terraform modules into one repo, their terraform configuration into a separate one, their application in another, and their helm configuration in yet another. Then if you want to add your helm configuration to ArgoCD, it’s yet another PR. I’m not a proponent of large, all encompassing mono-repos⁶, but some amount of grouping can be sensible.

If everything is grouped by framework or tool, rather than how it’s commonly interacted with, human intervention is almost always required. Let’s say I have to change an application. Modifying it it to consume a new external service. I’ll potentially have to open four pull requests and merge them in a very specific order.

build or modify a terraform module
implement a change in terraform
modify the application
change the helm chart

Sometimes more are required if it is a multi-step change or we have to wait for cloud provisioning to finish.

It’s easy to get the dependency ordering wrong and all too common to see ’no-op’ PRs created with the sole purpose of triggering another CI after something fails. Not to mention it’s completely mundane and repetitive. I’m not saying this workflow is completely broken, the deployments themselves are reproducible which is great. But calling this automated is disingenuous as there are a lot of repetitive tasks that could easily go away with a little bit of up-front planning.

But planning is hard to do if you working fast and conflate your design and implementation phases. Figuring it out as you go. It’s a lot “easier”, well at first anyways, to write a bunch of little things that you can reason about without a clear big picture. But the trade off is now you’ve de-coupled everything and will need to shuffle data in and out of your little units⁷. Which typically winds up being a manual process or some really brittle automation, which is just a manual process in disguise.

And again with automation

If we look back to my personal three items

Increased throughput or productivity

With many cloud provisioning tools the initial lift, learning curve, cognitive load, or whatever you want to call it is lower on the front end. Getting started. But managing a lot of the changes in many locations becomes more difficult to orchestrate.

This might be fine for small outfits or even startups, who’s whole business model is move fast while VCs more-or-less throw bologna at the wall to see what sticks. Many of these companies will never have large scale infrastructure. They don’t need to fully automate. Or many of the people in that situation won’t be around to deal with the fallout. They’re either software engineers white-knuckling it until an infra or DevOps person is hired, will have moved on to the next role, or the company will no longer exist

I guess that might reflect why everything is geared towards quick initial lift at the moment.

Relieving humans of monotonously repetitive work

Man, the number of PRs and ticky-tack little changes that have to be opened to accomplish anything at some organizations is astounding. The copy-pasting of boilerplate. Or even worse, tracking down and modifying said boiler plate. All dreadfully boring, all dreadfully common. It’s all too easy to design a workflow that is convoluted.

Increased human freedom to do other things

It’s very common for a DevOps engineer to basically be a glorified operator. Help tickets and slack messages to debug convoluted CI/CD pipelines, coordinating lots of repetitive changes that need to be merged in a specific order, tracking down old boiler plate that needs to be modified, and anything else mentioned above. It’s a time sink, and these aren’t adding value to organizations. Rather than manually orchestrate “automated” processes, these individuals would be better served solving other problems for the business.

But it’s not all fire and brimstone

It wasn’t my intent to come across all doom-and gloom and bad mouth a bunch of tools. To clarify; I like these tools. I use them daily. I reach for them when designing new things. In terms of automation as an industry, we’re at the highest point we’ve ever been. More people than ever have access to tools that help them automate tasks. Not to mention there is a network effect of steering everyone towards the same path⁸. Even if it’s not straightforward to achieve the levels of automation previous tools afforded, it’s not an apples-to-apples comparison. More is possible now.

I just wanted to say that it can be better. Those of us that automated it then are automating it now and we have opinions.

If you have an opinion on any of these tools, think I was completely off-base, have a correction, or would like to highlight additional tooling I would love to hear from you. contact at pallissard dot net

these are how salt’s states or puppet’s manifests are designed ↩︎
Ok, you can actually do this with some of the terraform tools, but it’s not as straightforward. ↩︎
I’d actually been complaining about it for years before I heard the official term “DRY” mentioned. ↩︎
or is it here?, it’s been unclear for a while, but it looks like they got it sorted out ↩︎
And for a lot of things, this makes sense. ↩︎
I lied, actually I am, but I’m pragmatic. Sometimes it just doesn’t make sense. ↩︎
Not to mention the fact that if you don’t have a clear picture you’re certainly not enforcing standards or infrastructure requirements with any sort of technical guard-rail. ↩︎
Yeah, this has a lot of downsides as well but I think it’s overwhelmingly a net positive. ↩︎

Cloud Network Design: Vendor Agnostic Network Config

2022-08-14

Taking configuration transformation and separation a step further

In a previous post I’d defined a simple VPC peering based network solution. There I highlighted how I prefer to separate declarative configuration from logic; defining the data in terms of how people interact with it on a day-to-day basis rather than how it’s used in code to interact with machines.

In that post I’d mentioned how one could imagine defining abstract interfaces to handle the idiosyncrasies between vendors. If you haven’t read that, feel free to skip over my network musings, but at least go back and familiarize yourself with the configuration format.

If you want to see the source for this you can find it here¹

Since we’re already transforming our configuration to a more automation-friendly form there is no reason we can’t abstract the vendor specific details away. The interface for subnets and their definition could look like so.

export interface SubnetData {}
export type SubnetDefinition<T extends SubnetData> = PartialRecord<
  Subnets,
  // Subnets are definied in our config.  This enforces valid names
  T
>;
export interface NetworkData<T> {
  subnets: SubnetDefinition<T>;
}

export type NetworkConfig<T, N extends NetworkData<T>> = Record<
  Vpcs,
  // Vpcs are definied in our config.  This enforces valid names
  N
>;

Given this, we can extend the interfaces; ensuring that our datastructures between cloud vendors are identical while sweeping the vendor specific configuration under the rug.

Config transformation examples

As we did previously, we’ll take a configuration that I like to work with directly and transform it to something easier to program with. We’ll add/transform to the vendor specific options as we go.

GCP specific interfaces

interface S extends SubnetData {
  config: {
    project: string;
    ipCidrRange: string;
    gatewayAddress: string;
  };
}
interface N<S> extends NetworkData<S> {
  project: string;
}

export class GcpNetworkingConfig implements NetworkDefinition {
  topo = topo;
  peers = peers;
  firewall = firewall;
  config: NetworkConfig<S, N<S>>;

  constructor(project: Project) {
    const l = getKeys(this.topo).map((j) => {
      const subnets = this.topo[j].subNets;
      return {
        [j]: {
          project: project.name,
          subnets: getKeys(subnets)
            .map((k) => {
              return {
                [k]: {
                  config: {
                    project: project.name,
                    ipCidrRange: subnets[k].cidr,
                    gatewayAddress: subnets[k].gateway,
                  },
                },
              } as SubnetDefinition<S>;
            })
            .reduce((obj, i) => {
              return { ...obj, ...i };
            }),
        },
      } as NetworkConfig<S, N<S>>;
    });
    this.config = l.reduce((obj, i) => {
      return { ...obj, ...i };
    });
  }
}

AWS

interface S extends SubnetData {
  config: {
    cidrBlock: string;
  };
}

interface N<S> extends NetworkData<S> {
  cidrBlock: string;
}

export class AwsNetworkingConfig implements NetworkDefinition {
  topo = topo;
  peers = peers;
  firewall = firewall;
  config: NetworkConfig<S, N<S>>;

  constructor() {
    const l = getKeys(this.topo).map((j) => {
      const subnets = this.topo[j].subNets;
      return {
        [j]: {
          cidrBlock: this.topo[j].cidr,
          subnets: getKeys(subnets)
            .map((k) => {
              return {
                [k]: {
                  config: {
                    cidrBlock: subnets[k].cidr,
                  },
                },
              } as SubnetDefinition<S>;
            })
            .reduce((obj, i) => {
              return { ...obj, ...i };
            }),
        },
      } as NetworkConfig<S, N<S>>;
    });
    this.config = l.reduce((obj, i) => {
      return { ...obj, ...i };
    });
  }
}

The implementation portion

This still leaves a fair amount of duplication/boilerplate between the implementations as you can see.

examples

AWS

export class AwsNetwork extends Construct {
  readonly config: NetworkingConfig;
  readonly vpcs: Record<Vpcs, vpc.Vpc>;
  readonly subnets: Record<Vpcs, Record<Subnets, vpc.Subnet>>;
  readonly peers: PartialRecord<
    Vpcs,
    PartialRecord<Vpcs, vpc.VpcPeeringConnection>
  >;
  readonly firewalls: Record<
    Vpcs,
    Record<Direction, Record<string, vpc.NetworkAclRule[]>>
  >;

  constructor(
    scope: Construct,
    name: string,
    config: NetworkingConfig,
  ) {
    super(scope, name);
    this.config = config;
    this.vpcs = this.networks(config);
    this.subnets = this.subnetworks(config);
    this.peers = this.networkspeering(config);
    this.firewalls = this.firewall(config);
  }
[snip]

GCP

export class GcpNetwork extends Construct {
  readonly config: NetworkingConfig;
  readonly vpcs: Record<Vpcs, ComputeNetwork>;
  readonly subnets: Record<Vpcs, Record<Subnets, ComputeSubnetwork>>;
  readonly peers: PartialRecord<
    Vpcs,
    PartialRecord<Vpcs, ComputeNetworkPeering>
  >;
  readonly firewalls: Record<
    Vpcs,
    Record<Direction, Record<string, ComputeFirewall>>
  >;
  constructor(
    scope: Construct,
    name: string,
    config: NetworkingConfig,
  ) {
    super(scope, name);
    this.config = config;
    this.vpcs = this.networks(config);
    this.subnets = this.subnetworks(config);
    this.peers = this.networkspeering(config);
    this.firewalls = this.firewall(config);
  }
[snip]

We could implement a class that takes an abstract type instead of the actual constructs like vpc.Subnets, ComputeSubnet, etc. This seems like it would be doable but would take at least

The class itself would have to accept a function from the GCP/AWS implementation
modifications to the type that describes our transformed configuration
modifications to the constructors that generate the config

But at that point abstracting may cost more than some boilerplate. I’ll think about it some more. If you have thoughts I’d love to hear them.

md5sum, gpg signature ↩︎

Aws cli v2 installation garbage

2022-07-21

Broke-ass way of shipping software

I feel like the aws-cli team is pretty ignorant of standard conventions when it comes to shipping software to end users. As usual, I have opinions.

installer fails but not really

 ---> Running in 66a1029f66b9
/aws/dist/aws: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
You can now run: /usr/local/bin/aws --version

Super cool, error but exiting zero. We can see right in the installer script that they don’t check return codes on subshells.

set_global_vars() {
  ROOT_INSTALL_DIR=${PARSED_INSTALL_DIR:-/usr/local/aws-cli}
  BIN_DIR=${PARSED_BIN_DIR:-/usr/local/bin}
  UPGRADE=${PARSED_UPGRADE:-no}

  EXE_NAME="aws"
  COMPLETER_EXE_NAME="aws_completer"
  INSTALLER_DIR="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )" 
  INSTALLER_DIST_DIR="$INSTALLER_DIR/dist"
  INSTALLER_EXE="$INSTALLER_DIST_DIR/$EXE_NAME"
  AWS_EXE_VERSION=$($INSTALLER_EXE --version | cut -d ' ' -f 1 | cut -d '/' -f 2)
[snip]

shipping a dynamically linked binary wrapper

It’s a damned bit of python code so you get cross-platform for free with the notable exceptions of any c-extensions that have non-portable implementations. Despite that they ship with a wrapper that is dynamically linked.

~  file /aws/dist/aws
/aws/dist/aws: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=1
da3a1d77c7109ce6444919f4a15e7e6c63d02fa, stripped

This means that the machine you run it on has to have the same shared libraries on as the machine that built it. Requiring shared libraries in-and-of itself is no big deal, this is how operating systems have worked for years and years. The main issue with this is that they expect glibc to be available. This completely ignores one of the most popular container platforms alpine, which uses muslc. If you have an upstream image that uses alpine and you need to add the aws-cli to it you’re out of luck. You have to either rebuild the upstream image with a glibc environment as the source, or futz about and get the awscli working in alpine.

Why they couldn’t just statically link this wrapper script is beyond me. Many languages compile statically linked binaries by default¹. Better yet, if they had provided a source distribution of their code we could plumb in a patch at build time to make the cli work with musl.²

Making this go

There are numerous threads and pages on this topic but variants of this SO answer seem to be the most popular. I opted for the self-hosted, poor-mans way of building this.

Remember that error above? error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory

~  mkdir kludge
# docker build (Dockerfile below)
# a wild error appears
~  cp /usr/lib64/libdl.so.2 kludge/
~  docker build -t test .

Wash, rinse, repeat for each shared library.

It wound up looking like this

The libraries. Not to bad, took all of 2 minutes.

~  ls kludge
ld-linux-x86-64.so.2  libc.so.6  libdl.so.2  libm.so.6  libpthread.so.0  librt.so.1  libutil.so.1  libz.so.1

And we’ll need a wrapper script so we can set LD_LIBRARY_PATH. To inform the linker of the shared library location when the binary is evoked.

~  cat bin/aws
#!/usr/bin/env bash

main(){
  export LD_LIBRARY_PATH=/lib64
  awscli "$@"
  exit $?
}

main "$@"

And we muck with the LD_LIBRARY_PATH during the install as well, taking care to unset it when done so our main application doesn’t run into a bunch of symbol-related issues itself.

~  cat Dockerfile
from alpine
run apk add curl bash
run mkdir /lib64      # musl doesn't use /lib64 by default
copy kludge/ /lib64/

run curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
		unzip awscliv2.zip && \
		rm awscliv2.zip

env LD_LIBRARY_PATH=/lib64
run /aws/install && mv /usr/local/bin/aws /usr/local/bin/awscli
copy bin/aws /usr/local/bin/aws
env LD_LIBRARY_PATH=

and there you have it

One self-hosted kludgy solution. Garbage in, garbage out.

~  docker build -t test .
[snip]
~  docker run -it test aws --version
aws-cli/2.7.17 Python/3.9.11 Linux/5.18.9-arch1-1 exe/x86_64.alpine.3 prompt/off

Which is wasteful in terms of memory and disk, but portable. ↩︎
This is fairly common for alpine builds ↩︎

Cloud Network Design: VPC peering

2022-07-08

Design considerations in the physical datacenter

In the physical datacenters that I’ve worked in we often spear-headed large initiatives to improve east-west traffic. Racks in the same row communicating with each other degrading performance in racks unaffiliated with the traffic in question; either due to hair-pinning or a distribution switch that just wasn’t up for the task.

From a pure bandwidth/latency perspective it would be ideal if every node ¹ would have a direct connection to every other node. If you set aside the scaling issues one might encounter with immense route tables, the sheer port count, and cable management headaches for a moment, this would be ideal;

Traffic is isolated to participating parties. You can’t effect your neighbor.
Bandwidth is high. Links aren’t shared.
Latency is as low. There are no hops
The network is robust. If one path goes down only two nodes are affected. And even then, only a subset is affected.

Since a we can’t cable every server to every server, we typically achieve some of the similar benefits that by moving to a spine and leaf architecture (a form clos networking) with ECMP. We get the following benefits;

Improving latency by reducing the number of hops
Improving overall throughput by removing bottlenecks and leveraging active-active links
Scaling is a simple operation, you add more switches.

To the cloud

Despite the fact that the cloud allows infrastructure to be stood up quickly with pre-existing automation tools, one still has to take into account the same design considerations of a physical data center. Maybe your application is a simple crud app that only needs to speak to external customers and north-south is all you really care about. But maybe you have an application that has a large persistence layer, or needs to share an inordinate amount of state.

Since we’re in the cloud, switch ports and cables are no longer problems we have to deal with. There are limits² to peering, particularly the number of peering connections. The limits do set an upper bound, but even with an upper bound, connecting all of these traditionally would be still be tedious and a management headache. Luckily for us we can automate this process with modern IaC tools and frameworks. Meaning, we can get several of the aformentioned benefits with a very simple design and minimal upkeep.

A side note on AWS’s transit gateway

Back in 2018, AWS released the transit gateway which is generally speaking, a hub and spoke architecture. While it has it’s use cases, and I’m sure many are bound to disagree with me, it seems like it’s a step back in a few ways.

centralized routing and firewall rules
throughput limitations

Not to mention, they did without it for what? over a decade? GCP still doesn’t have a 1-to-1 offering like the transit gateway ³. The TGW feels like it was crafted for management via the AWS console.

While transit gateways have a use-case and place. VPC peering has a lot to offer, especially when paired with some custom automation.

Separating all of the configuration components such as routing and firewall rules
- It’s much harder to make a site-wide mistake when managing separate rules from a central place
Improving overall throughput
With automation we avoid the headache of managing all of these peers.

That said there are limitations;

We don’t have transitive peering, so we are forced to go all in on a mesh⁴
As stated earlier the number of peers per VPC have an upper bound

Building a solution

I’ll step through a POC using cdktf and GCP. If you want to see the source for this you can find it here⁵

While there is an initial lift in coding, particularly between transforming your configuration into the form useable by terraform, the day to day maintenance burden is lower due to a sane configuration in a central location. The whole get shit done, MVP, mentality of work is often used by folks as an excuse to cut corners.

Rather than quickly throw up something that with off the shelf automation tools with little regard for future flexibility; why not just come up with a simple, maintainable in-house design?

NOT EVERYTHING HAS TO BE A COMMUNITY SUPPORTED MODULE. In fact I’d argue that for your foundational layers of infrastructure it can be worth your while to do some of the heavy lifting yourself.

You have a complete understanding of the dependencies
You control all future changes
You probably only need a subset of functionality so you can keep your implementation simple
You aren’t at the mercy of any upstream changes.

A contrived example

Defining your configuration, aka a schema

Rather than think about how the tools available to you expect their format defined⁶, I prefer to think in terms of how it would be preferable to modify and manage. Less “how do I make this work” and more “how do I want this to work”

In networks I think in terms of topographies;

And with CDKTF we can enforce an actual type on our configuration.

type Cidr = string;
type Gateway = string;
type VpcAllocation = {
  subNets: Record<Subnets, { cidr: Cidr; gateway: Gateway }>;
};
type Topo = Record<Vpcs, VpcAllocation>;

readonly topo: Topo = {
  infra: {
    subNets: {
      app: {
        cidr: '10.92.0.0/22',
        gateway: '10.92.0.1',
      },
      db: {
        cidr: '10.92.4.0/24',
        gateway: '10.92.4.1',
      },
    },
  },
  management: {
    subNets: {
[snip]

peers I think of mappings

Now, this looks like it could be cut down further as there are always two ends of a peering. One would be tempted to specify the peering once. But by specifying the peering on each end;

It’s easy to look up a given VPC and see what it’s peered with.
We don’t have to come up with a policy/system for determining which network the peering is specified. (ex. Alphabetical? what if we add a new network that upsets that mix?)
There are technically two parts to a peer and we’re specify them both explicitly. IMO this is easy to reason about.

readonly peers: Peers = {
  management: ['infra', 'sales'],
  infra: ['management'],
  sales: ['management'],
};

and in firewall rules I think of traditional firewall chains

In the firewall rules I am using terms that closely map to the GCP terraform provider, although I have added an additional sourceNetworks field that I can match on to pull cidr blocks out of the live config.

readonly firewall: Record<Vpcs, FwConfig> = {
  management: {
    ingress: [
      {
        name: "ping",
        allow: true,
        priority: 1000,
        protocol: 'icmp',
        sourceNetworks: {infra: 'app'} // here
      },
      {
        name: "catchall",
        allow: false,
        priority: 10001,
        protocol: 'all',
        sourceRanges: ["0.0.0.0/0"]
      },

    ],
    egress: [
      {
        name: "ping",
        allow: true,
        priority: 1000,
        protocol: 'icmp',
      },
      {
        name: "catchall",
        allow: false,
        priority: 10001,
        protocol: 'all',
      },
    ],
  },
  infra: {
    egress: [
      {
        name: "ping",
        allow: true,
[snip]

making our representation useful.

types to leverage

Now that we’ve defined the types that are easy to modify.

We can define the type that is easy to work with in the language.

export type SubnetConfig = PartialRecord<
  Subnets,
  { config: ComputeSubnetworkConfig }
>;

export type NetworkConfig = Record<
  Vpcs,
  {
    project: string;
    subnets: SubnetConfig;
  }
>;

And then we can wrap it all up in constructor.

Allowing us to have our cake and eat it too. We have a type that’s easy to define, and a type that’s easy to work with.

  constructor(project: Project) {
    const l = getKeys(this.topo).map((j) => {
      const subnets = this.topo[j].subNets;
      return {
        [j as Vpcs]: {
          project: project.name,
          subnets: getKeys(subnets)
            .map((k) => {
              return {
                [k]: {
                  config: {
                    project: project.name,
                    ipCidrRange: subnets[k].cidr,
                    gatewayAddress: subnets[k].gateway,
                  },
                },
              } as SubnetConfig;
            })
            .reduce((obj, i) => {
              return { ...obj, ...i };
            }),
        },
      } as NetworkConfig;
    });
    this.config = l.reduce((obj, i) => {
      return { ...obj, ...i };
    });
  }

using the representation

And now that our easy to maintain representation has been transformed to one that is a bit easier to program with all of our tasks become simple. We can even stash the return types fields in the object iteslf for easy consumption by other modules.

I won’t go on and on into the details of everything, if you want to see the nitty gritty see the source I linked to above.

networks

networks(i: GcpNetworkingConfig) {
  return getKeys(i.config)
    .map((v: Vpcs) => {
      const rv = new ComputeNetwork(this, v, {
        name: v,
        project: i.config[v].project,
        autoCreateSubnetworks: false,
        deleteDefaultRoutesOnCreate: true,
      });
      return { [v]: rv } as Record<Vpcs, ComputeNetwork>;
    })
    .reduce((obj, i) => {
      return { ...obj, ...i };
    });
}

and subnets

subnetworks(i: GcpNetworkingConfig) {
  return getKeys(i.config)
    .map((v: Vpcs) => {
      return {
        [v]: getKeys(i.config[v].subnets)
          .map((s: Subnets) => {
            const c = assert(i.config[v].subnets[s]).config;
            return {
              [s]: new ComputeSubnetwork(this, `${v}-${s}`, {
                ...c,
                name: `${v}-${s}`,
                network: this.vpcs[v].name,
              }),
            } as Record<Subnets, ComputeSubnetwork>;
          })
          .reduce((obj, i) => {
            return { ...obj, ...i };
          }),
      } as Record<Vpcs, Record<Subnets, ComputeSubnetwork>>;
    })
    .reduce((obj, i) => {
      return { ...obj, ...i };
    });
}

and firewalls

ok, admittedly this one’s a bit beefier and shows some of typescripts warts. ⁷

firewall(i: GcpNetworkingConfig) {
  return getKeys(i.firewall)
    .map((v) => {
      return getKeys(i.firewall[v])
        .map((direction) => {
          const ad = (i: boolean): 'allow' | 'deny' => {
            return i == true ? 'allow' : 'deny';
          };
          return i.firewall[v][direction]
            .map((j) => {
              const sourceRanges =
                j.sourceRanges != undefined ? j.sourceRanges : [];

              const sourceNetworks =
                j.sourceNetworks != undefined
                  ? getKeys(j.sourceNetworks).map((net) => {
                      return this.subnets[net][
                        // typechecker ain't that great
                        assert(assert(j.sourceNetworks)[net])
                      ].ipCidrRange;
                    })
                  : [];
              return {
                [v]: {
                  [direction]: {
                    [j.name]: new ComputeFirewall(
                      this,
                      `${v}-${direction}-${ad(j.allow)}-${j.name}`,
                      {
                        name: `${v}-${direction}-${j.name}`,
                        project: this.vpcs[v].project,
                        [ad(j.allow)]: {
                          protocol: j.protocol,
                          ports: j.ports,
                        },
                        priority: j.priority,
                        sourceTags: j.sourceTags,
                        sourceRanges: [
                          ...sourceRanges,
                          ...sourceNetworks,
                        ],
                        network: this.vpcs[v].id,
                        direction: direction.toUpperCase(),
                        logConfig: {
                          metadata: 'INCLUDE_ALL_METADATA',
                        },
                      },
                    ),
                  },
                },
              } as Record<
                Vpcs,
                Record<Direction, Record<string, ComputeFirewall>>
              >;
            })
            .reduce((obj, i) => {
              return { ...obj, ...i };
            });
        })
        .reduce((obj, i) => {
          return { ...obj, ...i };
        });
    })
    .reduce((obj, i) => {
      return { ...obj, ...i };
    });
}

You get the idea

So yeah, peering is good and typescript is alright for this sort of thing. It feels a bit heavier than dealing with HCL when yout take into account the dev environment and initial setup. But dealing with it on a day to day basis is less painful despite the fact that the tools in TS/JS ecosystem are resource hogs.

Additionally, I could imagine defining an abstract class with methods like network, firewall, etc that accept a config block and set up the proper constructs. Then you could have a class for each cloud provider that implemented the methods, created the proper contstructs, and mapped the fields properly. Keeping your config somewhat agnostic between the providers. Naturally there’d be edge cases all over the place, but it’s a good thought exercise for practicing clean design.

In this context, when I say node I mean server, appliance, anything. Not network nodes in the strict sense. ↩︎
Currently it defaults to 50 for AWS and 25 for gcp. Although those limits can be increased. ↩︎
Although it’s likely not needed since the way GCP segregates infrastructure by projects is a lot less heavy handed than how AWS namespaces via account. The account is such a brutal namespace. ↩︎
Well not exactly, we’re not going to peer VPCs that don’t need to talk to each other. Satisfying security requirements and helping keep us under the VPC peering limits. ↩︎
md5sum, gpg signature ↩︎
Not that there isn’t at time and place to think of your output format ↩︎
While my approach could likely be improved, not being able to type narrow via ternary is quite a shame. ↩︎

Limiting application resources

2022-06-27

I run a lot of hardware that’s well over a decade old. Up until last year I exclusively used an HP USDT as my daily driver. While it often works without issue, many “modern” applications are resource intensive. Often leading to cpu starvation or memory contention. In order to keep the desktop environment afloat in these situations I’ve been wrapping commands in cgroups using systemd to enforce memory limits on them. Even though I have had a modern laptop for the past year, I still enforce these same limits¹. I figured I’d share my ~~hacky~~ convenient zsh aliases for performing such things.

This isn’t so much an actual post as it is a dump of some aliases.

Dispatching commands

I’m not a complete systemd fanboy, but one of the nice things is how process groups have a pair of abstractions layers, the slice and the unit, thrown on top of them.² This lets you group commands into units, and units into slices. You can carve up a slice as a category; imposing a limit on the resource sums for all units in said slice. Then each unit can use individual resource limits. If a unit hits its individual limit, it will be throttled, start swapping, or be killed individually. But if the sum of all the units in a slice hit the slice limits, they will be penalized as a whole.

User slices follow the following format³

~  cat ~/.config/systemd/user/browsers.slice
[Unit]
Description=browsers
DefaultDependencies=no
Before=slices.target
Requires=system.slice
After=system.slice

[Slice]
MemoryHigh=10G
MemoryMax=11G
MemorySwapMax=1024M

I wrap it in a dispatching function, mainly so I can plumb in changes centrally.

function dispatch {

	if [[ $USER == "root" ]]; then
		command "$binary" "$@"
		return $?
	fi

	declare args=(
		--user
		--same-dir
		-p IOAccounting=yes
		-p MemoryAccounting=yes
		-p TasksAccounting=yes
	)
	while (($#)); do
		case "$1" in
			-c)
				args+="-p"
				args+="CPUWeight=$2"; shift 2
				;;
			-mm)
				args+="-p"
				args+="MemoryMax=$2"; shift 2
				;;
			-mh)
				args+="-p"
				args+="MemoryHigh=$2"; shift 2
				;;
			-s)
				args+="-p"
				args+="MemorySwapMax=$2"; shift 2
				;;
			--scope)
				args+=--scope; shift
				;;
			--slice)
				args+="--slice=$2"; shift 2
				;;
			--name)
				name=$1; shift 2
				;;
			-P)
				args+=-P; shift
				;;
			--binary)
				[ -z "$name" ] || name=$2
				binary="$2"; shift
				;;
			*)
			break
		esac
	done
	systemd-run $args "$@"  2> >(>&2 grep -vE 'Running.*as unit:')
}

Now, this works all well and good from a terminal. But a lot of processes call fork/execv and for those I drop a wrapper in a ~/bin directory that I’ve added to my path.

~  cd ~/bin
~  cat .dispatch/template
#!/usr/bin/env zsh
# redirecting source to dev null is important when you're parsing output
source ~/.zshrc >/dev/null 2>/dev/null 
"${ZSH_ARGZERO##*/}" $@
~  ln -s .dispatch/template firefox

This is a must if you want to wrap commands from your launcher or some of your lsp evoked commands. ⁴ That’s basically it for the mechanics of the thing.

A little more nuance

While I’m on the topic of lsp/development environments; so many commands use node that I wrap those separately.

function node {
	case "$1" in
		*"typescript-language-server"*)
			dispatch --name tsserver \
				--scope \
				--slice node \
				-c 3000 \
				-mh 3.5G \
				-mm 3.6G \
				-s 128M \
				--binary /usr/bin/node "$@"
			;;
		*"pyright"*)
			dispatch --name pyright \
				--scope \
				--slice pythonlsp.slice \
				-c 100 \
				-mh 2000M \
				-mm 2048M \
				-s 48M \
				--binary /usr/bin/node "$@"
			;;
		*"tsc"*)
			dispatch \
				--name tsc \
				--scope \
				--slice node \
				-c 135 \
				-mh 2000M \
				-mm 2048M \
				-s 24M \
				--binary /usr/bin/node "$@"
			;;
		*"cdktf"*)
			dispatch \
				--name cdktf \
				--scope \
				--slice node \
				-c 35 \
				-mh 3G \
				-mm 3G \
				-s 24M \
				--binary /usr/bin/node "$@"
			;;
		*"eslint_d"* | *"core_d"*)
			dispatch \
				--name eslint_d \
				--scope \
				--slice node \
				-c 200 \
				-mh 4.3G \
				-mm 4.5G \
				-s 24M \
				--binary /usr/bin/node "$@"
			;;
		*"eslint"*)
			dispatch \
				--name eslint \
				--scope \
				--slice node \
				-c 300 \
				-mh 2.5G \
				-mm 2.6G \
				-s 24M \
				--binary /usr/bin/node "$@"
			;;
		*"prettier-eslint"*)
			dispatch \
				--name prettier-eslint \
				--scope \
				--slice node \
				-c 300 \
				-mh 500M \
				-mm 512M \
				-s 12M \
				--binary /usr/bin/node "$@"
			;;
		*"prettier"*)
			dispatch \
				--name prettier \
				--scope \
				--slice node \
				-c 300 \
				-mh 500M \
				-mm 512M \
				-s 12M \
				--binary /usr/bin/node "$@"
			;;
		*"diagnostic-languageserver"*)
			dispatch \
				--name diagnostic-languageserver \
				--scope \
				--slice node \
				-c 90 \
				-mh 1024M \
				-mm 1024M \
				-s 12M \
				--binary /usr/bin/node "$@"
			;;
		*"npm"*)
			dispatch \
				--name npm \
				--scope \
				--slice node \
				-c 35 \
				-mh 1500M \
				-mm 1572M\
				-s 72M \
				--binary /usr/bin/node "$@"
			;;
		*)
			dispatch \
				--name node-generic \
				--scope \
				--slice node \
				-c 35 \
				-mh 500M \
				-mm 512M \
				-s 12M \
				--binary /usr/bin/node "$@"
			;;
	esac
}

I match python similarly ⁵

python () {
	if [ -n  "$VIRTUAL_ENV" ]; then
		source $VIRTUAL_ENV/bin/activate
	fi
	case "$1" in
		*"jedi"*)
			dispatch --name jedi-language-server \
				--scope \
				--slice pythonlsp.slice \
				-c 100 \
				-mh 2000M \
				-mm 2048M \
				-s 48M \
				--binary python "$@"
			;;
		*)
			dispatch \
				--name python-generic \
				--scope \
				--slice pythonlsp.slice \
				-c 35 \
				-mh 500M \
				-mm 512M \
				-s 12M \
				--binary python "$@"
			;;
	esac
}

Introspection

You can check on the status of it with systemctl --user status browsers.slice. Example output is as follows. Notice how there are two scope units nested under the slice. The firefox and chrome pids are nested under their respective scope.

● browsers.slice - browsers
     Loaded: loaded (/home/matt/.config/systemd/user/browsers.slice; static)
    Drop-In: /home/matt/.config/systemd/user.control/browsers.slice.d
             └─50-MemoryHigh.conf, 50-MemoryMax.conf
     Active: active since Wed 2022-06-22 05:54:52 AKDT; 5 days ago
      Until: Wed 2022-06-22 05:54:52 AKDT; 5 days ago
      Tasks: 753
     Memory: 3.5G (high: 10.1G max: 10.0G swap max: 1.0G available: 6.4G)
        CPU: 10h 24min 23.110s
     CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice
             ├─run-rd666fcddb4584959b3e72ec7422f707f.scope
             │ ├─2707939 /usr/lib/firefox/firefox
             │ ├─2708074 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 9513 -prefMapSize 266077 -appDir /usr/lib/fir>
             │ ├─2708100 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 9631 -prefMapSize 266077 -jsInitLen 277128 -parent>
             │ ├─2708143 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 14640 -prefMapSize 266077 -jsInitLen 277128 -paren>
             │ ├─3369479 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 18146 -prefMapSize 266077 -appDir /usr/lib/fi>
             │ ├─3374574 /usr/lib/firefox/firefox -contentproc -childID 161 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3473440 /usr/lib/firefox/firefox -contentproc -childID 168 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3585007 /usr/lib/firefox/plugin-container /home/matt/.mozilla/firefox/i3w1cbew.default/gmp-widevinecdm/4.10.2449.0 2707939 true gmp>
             │ ├─3619513 /usr/lib/firefox/firefox -contentproc -childID 195 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3619611 /usr/lib/firefox/firefox -contentproc -childID 196 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3619778 /usr/lib/firefox/firefox -contentproc -childID 197 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3768700 /usr/lib/firefox/firefox -contentproc -childID 210 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3774713 /usr/lib/firefox/firefox -contentproc -childID 211 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3887801 /usr/lib/firefox/firefox -contentproc -childID 213 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3930854 /usr/lib/firefox/firefox -contentproc -childID 217 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3933460 /usr/lib/firefox/firefox -contentproc -childID 218 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3933865 /usr/lib/firefox/firefox -contentproc -childID 219 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ ├─3935821 /usr/lib/firefox/firefox -contentproc -childID 220 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             │ └─3936239 /usr/lib/firefox/firefox -contentproc -childID 221 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
             └─run-re6a64a3f61e9425280a6cc758d955648.scope
               ├─3927358 "/usr/lib/chromium/chromium --enable-features=UseOzonePlatform --ozone-platform=wayland 
               ├─3927371 /usr/lib/chromium/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler "--database=/home>
               ├─3927373 /usr/lib/chromium/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler "--database=>
               ├─3927377 "/usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --enable-crashpad --crashpad-handler-pid=3927371 --enable-cra>
               ├─3927378 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
               ├─3927380 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
               ├─3927405 "/usr/lib/chromium/chromium --type=gpu-process --ozone-platform=wayland --enable-crashpad --crashpad-handler-pid=3927371 -->
               ├─3927406 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-t>
               ├─3927411 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-t>
               ├─3927482 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >
               ├─3927483 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >
               └─3927558 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >

Jun 22 05:54:52 matt-gen-laptop-p01 systemd[809]: Created slice browsers.

I use the brittle peek alias often

function peek {
	[ -n "$1" ] || bail 1 "specify a slice"
	fid=()
	while read unit; do
	fid+=r"$unit"
	done < <(systemctl --user status "$1" | grep scope | cut -f 2- -d r)
	systemctl --user status $fid 
}

which in turn relies on bail, but that isn’t a hard requirement

function bail {
	if [ -z "$1" ]; then
		printf "no exit code returned\n"
		return 1
	elif [ "$1" -ne 0 ]; then
		 [[ -z "$2" ]] && printf "failed\n" || printf "%s\n" "$2"
	fi
	return "$1"
}

A variant of peek obcess

function obcess {
  [ -n "$1" ] || bail 1 "specify a slice"
  watch 'zsh -c "source /home/matt/.zshrc >/dev/null 2>/dev/null && peek '$1'"'
}

Peek in action.

~ peek browsers.slice
● run-rd666fcddb4584959b3e72ec7422f707f.scope - /usr/bin/firefox
     Loaded: loaded (/run/user/10010/systemd/transient/run-rd666fcddb4584959b3e72ec7422f707f.scope; transient)
  Transient: yes Active: active (running) since Thu 2022-06-23 21:13:14 AKDT; 3 days ago
      Tasks: 620 (limit: 33435)
     Memory: 2.6G (high: 6.3G max: 6.3G swap max: 12.0M available: 3.6G)
        CPU: 8h 10min 52.728s
     CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice/run-rd666fcddb4584959b3e72ec7422f707f.scope
             ├─2707939 /usr/lib/firefox/firefox
             ├─2708074 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 9513 -prefMapSize 266077 -appDir /usr/lib/firef>
             ├─2708100 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 9631 -prefMapSize 266077 -jsInitLen 277128 -parentBu>
             ├─2708143 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 14640 -prefMapSize 266077 -jsInitLen 277128 -parentB>
             ├─3369479 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 18146 -prefMapSize 266077 -appDir /usr/lib/fire>
             ├─3374574 /usr/lib/firefox/firefox -contentproc -childID 161 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3473440 /usr/lib/firefox/firefox -contentproc -childID 168 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3585007 /usr/lib/firefox/plugin-container /home/matt/.mozilla/firefox/i3w1cbew.default/gmp-widevinecdm/4.10.2449.0 2707939 true gmplu>
             ├─3619513 /usr/lib/firefox/firefox -contentproc -childID 195 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3619611 /usr/lib/firefox/firefox -contentproc -childID 196 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3619778 /usr/lib/firefox/firefox -contentproc -childID 197 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3768700 /usr/lib/firefox/firefox -contentproc -childID 210 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3774713 /usr/lib/firefox/firefox -contentproc -childID 211 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3887801 /usr/lib/firefox/firefox -contentproc -childID 213 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3930854 /usr/lib/firefox/firefox -contentproc -childID 217 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3933460 /usr/lib/firefox/firefox -contentproc -childID 218 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3933865 /usr/lib/firefox/firefox -contentproc -childID 219 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             ├─3935821 /usr/lib/firefox/firefox -contentproc -childID 220 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
             └─3936239 /usr/lib/firefox/firefox -contentproc -childID 221 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>

Jun 23 21:13:14 matt-gen-laptop-p01 systemd[809]: Started /usr/bin/firefox.

● run-re6a64a3f61e9425280a6cc758d955648.scope - /usr/bin/chromium
     Loaded: loaded (/run/user/10010/systemd/transient/run-re6a64a3f61e9425280a6cc758d955648.scope; transient)
  Transient: yes
     Active: active (running) since Mon 2022-06-27 16:09:22 AKDT; 12min ago
      Tasks: 123 (limit: 33435)
     Memory: 506.6M (high: 3.4G max: 3.4G swap max: 12.0M available: 2.9G)
        CPU: 3.823s
     CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice/run-re6a64a3f61e9425280a6cc758d955648.scope
             ├─3927358 "/usr/lib/chromium/chromium --enable-features=UseOzonePlatform --ozone-platform=wayland 
             ├─3927371 /usr/lib/chromium/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler "--database=/home/m>
             ├─3927373 /usr/lib/chromium/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler "--database=/h>
             ├─3927377 "/usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash>
             ├─3927378 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Linu>
             ├─3927380 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Linu>
             ├─3927405 "/usr/lib/chromium/chromium --type=gpu-process --ozone-platform=wayland --enable-crashpad --crashpad-handler-pid=3927371 --en>
             ├─3927406 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-typ>
             ├─3927411 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-typ>
             ├─3927482 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
             ├─3927483 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
             └─3927558 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>

Jun 27 16:

Sundries

thoughts on web browsers

I’d really like to see support in firefox for their tab categories. Each tab category can be restricted by a slice or unit. A clean implementation would probably require firefox speaking to systemd via dbus.

wrapping one-off commands

function wrap {
	limit=$1; shift;
	dispatch --name $1 \
		--scope \
		--slice wrap.slice \
		-c 50 \
		-mh $limit \
		-mm $limit \
		-s 1\
		--binary "$@"
}

killing everything in a slice

systemctl --user stop node.slice

checking for failed commands

systemctl --user failed

check all of the user processes and their slices

systemctl --user status

some of the other slices I use

~  ls ~/.config/systemd/user/*.slice
/home/matt/.config/systemd/user/browsers.slice
/home/matt/.config/systemd/user/nvim.slice
/home/matt/.config/systemd/user/tmux.slice
/home/matt/.config/systemd/user/compilers.slice
/home/matt/.config/systemd/user/ocamllsp.slice
/home/matt/.config/systemd/user/vpn.slice
/home/matt/.config/systemd/user/gopls.slice
/home/matt/.config/systemd/user/pythonlsp.slice
/home/matt/.config/systemd/user/wrap.slice
/home/matt/.config/systemd/user/node.slice
/home/matt/.config/systemd/user/shell.slice

Well except for browsers and node, those get a slightly higher limit on a newer machine. ↩︎
I’ve spent all of my career dealing with posix environments and a good chunk in HPC. I have to say that process groups could have been fleshed out a little more fully. That fact that abstraction layers or additional automation are needed to properly manage children is a testament to that. ↩︎
Now why does a web browser can require that much RAM? ↩︎
I’m looking at you tsc. ↩︎
I’m going to skip the disaster that is how I manage my virtual environments for now. ↩︎

Don't trust. Verify.

2022-06-14

tl;dr I was disappointed to find out that typescript isn’t very strongly typed. You still have to keep an eye out for any, the generic object, and type coercion.

At one of my more recent gigs, I was thrown into a Typescript environment via cdktf¹. Having dabbled in OCaml² and Haskell previously I was looking forward to having a strong, statically typed language to define infrastructure with. Two of my biggest gripes with many configuration management and IaC solutions are a lack of schema enforcement and the heavy use of templating. ³

Having a full type system allows for some really nice things like this.

type domains = 'pallissard.net' | 'matt.pallissard.net';
export type validEmail = `${string}@${domains}`;

export interface UserConfig {
  email: validEmail;
  firstName: string;
  lastName: string;

This affords one the opportunity to;

Separate their code from their configuration
Use the type system as a DSL of sorts for declaring configuration. Handling schema enforcement at build time.

Exactly what I would hope for from a language with a strong type system. However, I immediately ran into an eyebrow raiser. This compiles.

const parsed = JSON.parse('[1]')
const bar = 1 + parsed

In fact, not only does it compile. Type coercion trumps annotations. You can see with with the string representation of 11 in the output below. The annotations feel closer to type hints than actual annotations.

const parsed: number = JSON.parse('[1]')
const bar: number = 1 + parsed
const baz: number = parsed[0] + 1

11 2

We’re effectively seeing the Javascript bleed through with the any and the type coercion. I’m shocked⁴ that for how popular this language is, and for how vocal the proselytizers are, the type-checker isn’t catching these things. Things that people say you should use Typescript for. This is supposed to be it’s core competency, turning runtime errors into compile time errors. ⁵

Interestingly enough, if we change our example a little bit.

const parsed: number = JSON.parse('{"k": 1}')
console.log(parsed + 1, parsed.k + 1)

We do receive compile time error, but it’s for a reference to the field k. Not for the incompatible addition operation parsed + 1


foo.ts:2:32 - error TS2339: Property 'k' does not exist on type 'number'.

2 console.log(parsed + 1, parsed.k + 1)
                                 ~


Found 1 error.

But if we modify it a bit more, we once again escape the compiler errors

const parsed: number = JSON.parse('{"k": 1}')
console.log(parsed + 1, parsed["k"] + 1)

I’m not saying typescript is all bad.⁶. Sometimes you’re stuck needing to write something in-browser, or the tool you need is written in javascript. But if one is reaching for typescript just for the type system there are certainly better options out there.

Examples

I’ve had it pointed out by some that have grown up in javascript and have ~~Stockholm syndrome~~ never been exposed to other languages and paradigms, that “Typescript transpiles to Javascript and Javascript doesn’t have types so there are limitations”. ⁷ Do you know what also doesn’t have types? Machine code. You know what a whole gang of statically typed languages use as a compilation target? Machine code.

Here are some examples. Not only do we get compilation errors, for some it feels awkward to write them incorrectly due to the type annotations and how the language generally operates.

OCaml

If we try to do the same operation in OCaml

let () =
  let open Yojson.Basic.Util in
  let foo = Yojson.Basic.from_string "{\"k\": 1}" in
  print_int (foo + 1)

We get a compile time error instead of a runtime error

4 |   print_int (foo + 1)
                 ^^^
Error: This expression has type Yojson.Basic.t
       but an expression was expected of type int

But if we explicitly unpack and cast..

let () =
  (* no error handling *)
  let open Yojson.Basic.Util in
  let foo = Yojson.Basic.from_string "{\"k\": 1}" in
  print_int (
    to_int(
      member "k" foo
    )+1
  );
  print_newline ();

we get

Haskell

Again, like the typescript example attempts.

{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
module Main(main) where

import Prelude.Compat
import Data.Aeson
import Lib
import Control.Applicative (empty)

data Foo = Foo { k :: Integer }
        deriving (Show)

instance FromJSON Foo where
    parseJSON ( Object v )  = Foo <$>
                              v .: "k"
    parseJSON _             = empty


main :: IO ()
main = do
    let bar = decode "{\"k\":1}" :: Maybe Foo
    bar+1 {- Specifically this line -}

Unsurprisingly, a compile time error

    • Couldn't match type ‘Maybe’ with ‘IO’
      Expected: IO Foo
        Actual: Maybe Foo

but if we swap bar+1 out with some pattern matching

    case bar of
        Just(Foo(i)) -> print (i+1)

we get the expected output

java

Java’s a little different than the above. It’s similar to how you should actually handle this in typescript. You create an object, and instantiate an instance of your object with the json object.

import com.google.gson.Gson;

public class foo {
  private class Bar {
    public Integer k;
    public Bar(Integer i){
      this.k = i;
    }
  }
  public static void main(String[] args) {
    Gson json = new Gson();
    Bar bar = json.fromJson("{\"k\": 1}", Bar.class);
    System.out.println(bar+1);
  }
}

But that said, it still catches this, where typescript would not.

foo.java:13: error: bad operand types for binary operator '+'
    System.out.println(bar+1);
                          ^
  first type:  foo.Bar
  second type: int
1 error

Expect a separate post on that later. ↩︎
https://discuss.OCaml.org/t/multicore-OCaml-dec-2020-jan-2021/7225 ↩︎
I’m still puzzled as to why the C preprocessor is evil while go templating will save us all. ↩︎
Not really, popular != (good | sane) ↩︎
technically we didn’t even get a runtime error. ↩︎
I’m just saying that javascript and the entirety of it’s ecosystem is. ↩︎
I kid you not. ↩︎