
Recently I was working on a nodejs/express application to troubleshoot a memory leak. Those were the steps and tools I followed to identify the root cause.

## 1. Reproduce the issue

We had it in two modules, one a http server and the other a sqs consumer.

#### For the http server

Created a [k6](https://k6.io/) script for a small performance test:

```javascript
// script.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export default function () {
  const url = 'http://localhost:3000/test';
  const res = http.get(url);
  check(res, { success: (r) => r.status === 200 });
}
```

And ran it with:

```bash
# 100 users, 1 hour
k6 run -u 100 -d 1h script.js
```

#### For the SQS consumer

Built a send command (note: I am using [localstack](https://localstack.cloud/)):

```bash
# send.sh
AWS_REGION: "us-east-1" aws --endpoint-url=http://localhost:4566
    sqs send-message
    --region "us-east-1"
    --queue-url http://localhost:4566/000000000000/queue
    --message-body '{"my": "message"}'
```

and ran with with watch:

```bash
watch -n .1 ./send.sh
```

## 2. Next step, collect the data

Initially I was using node itself for generating it, e.g:

```typescript
import v8 from 'node:v8';
// ...
app.get('/heapdump', (req, res) => {
  const fileName = v8.writeHeapSnapshot();
  res.send({ fileName });
});
```

but we can also use chrome devtools for this (I am using [brave](https://brave.com/), but sames applies):

- run the app with the flag `--inspect`
- go to `chrome://inspect` / `brave://inspect`, connect
- go to `Memory` tab, click on `Take Heap Snapshot`

## 3. Analyze the data

That is the tricky part. Pick the initial snapshot, run the test, pick the final snapshot, and compare.

Try to correlate what you have there with the code. `Allocation on timeline` was also helpful for me.

I strongly recommend take a look on [this presentation](https://www.youtube.com/watch?v=hliOMEQRqf8) for detailed explanation and showcase:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hliOMEQRqf8?si=iQ3kfqgyt6Aa_TZU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Next?

Now is the time to apply a fix and re-run the tests.

We found a lot of [DerivedLogger](https://github.com/winstonjs/winston/blob/c69cdb0cec15a138e0b6e374501e027d1c39606c/lib/winston/create-logger.js#L36) in the memory. It was because the code was calling `winston.createLogger` on each call to enrich the log. We switched to [child](https://github.com/winstonjs/winston#creating-child-loggers) call and problem solved.

There was no need to use [clinic.js](https://clinicjs.org/) this time.



Some steps for finding/troubleshooting it

Memory Leak in nodejs/javascript


I just tried the new cloudflare worker. To create a sample and deploy there, you just need to run one single command:

```
pnpm create cloudflare@latest <project name>
```

... and of course, a lot of prompts.

I removed some dead code (editor config, prettier), added [biome](https://biomejs.dev/), added a [taskfile](https://taskfile.dev/) (I will forget the commands). Also played with [Hono](https://github.com/honojs/hono) - the web framework [suggested by Cloudflare](https://www.youtube.com/watch?v=H7Qe96fqg1M) and added an AI endpoint.

Pretty simple, reminds me the good old days with the free [Heroku](https://www.heroku.com/) tier.

For sure I will consider it for my pet projects.

My full code is available on [github](https://github.com/adamatti/learn-cf-worker).

Next, I will try it with [remix](https://remix.run/) for a full stack experience. If I got it right, it should be [super simple](https://hono.dev/examples/with-remix) (or [creating from scratch](https://developers.cloudflare.com/pages/framework-guides/deploy-a-remix-site/)).


Cloudflare workers


Hey, are you using [yarn](https://classic.yarnpkg.com)? Great tool from [Facebook](https://facebook.com), right?

But how are you installing it?

`npm install -g yarn`?

### I have a bad news to tell you...

> Yarn has not been distributed on npm since 2019 because of breaking changes that could impact deployments - Google AI

and

> Those versions entered maintenance mode in January 2020 - [yarn classic website](https://classic.yarnpkg.com/lang/en/docs/install)

### So, what should I do?

Need to consider a better/newer package manager.

It could be:

- [the new yarn version](https://yarnpkg.com/): as of today (09/06), it is on version 4.4.1
- [PNPM](https://pnpm.io/), the faster alternative
- [The NPM itself](https://www.npmjs.com/). The official tool. In 2024, it is faster than yarn classic

There are a lot of benchmarks out there, like:

- https://romanglushach.medium.com/comparing-npm-yarn-and-pnpm-package-managers-which-one-is-right-for-your-distributed-project-to-4d7de2f0db8e
- https://refine.dev/blog/pnpm-vs-npm-and-yarn
- https://www.dhiwise.com/post/pnpm-vs-npm-vs-yarn-which-javascript-package-manager

### How about me?

I am using [PNPM](https://pnpm.io/) in my latest projects, both pet and professional, but already had some issues with it.

It all case by case, but I strongly recommend to use [PNPM](https://pnpm.io/) for fast local builds and CI.


Yarn Classic


Ok, Make/Makefiles are great. I spoke about then in the [last post](/blog/posts/2024-04-08-makefile).

But what if:

- I want to break them in different files
- Use a better syntax (maybe yaml?)
- Cache the tasks, even the ones that don't generate artifacts (e.g. lint)
- put tasks and aliases close
- validate the task execution
- have global tasks to run in any folder (e.g. backup)
- dry run
- watch task

#### Taskfile to rescue

Just need to [install Task](https://taskfile.dev/installation/) (or go-task on [github](https://github.com/go-task/task)).

The [documentation](https://taskfile.dev/) is pretty simple IMHO, but I want to highlight some things that I do use in daily bases that I think it does a great difference.

## Show me the code

For this I will use the sample from [previous post](/blog/posts/2024-04-08-makefile) ([makefile here](https://github.com/adamatti/node-sample/blob/main/Makefile))

```yaml
# yaml-language-server: $schema=https://json.schemastore.org/taskfile.json
---
version: '3'

env:
  DATABASE_URL: postgres://postgres:sample@localhost:5432/sample

includes:
  docker-compose:
    aliases: [dc]
    taskfile: ./Taskfile-docker-compose.yaml
  prisma:
    aliases: [p]
    taskfile: ./Taskfile-prisma.yaml

tasks:
  default:
    desc: Show help
    aliases: [h]
    silent: true
    cmds:
      - task -l --sort alphanumeric

  clean:
    desc: Remove node modules
    aliases: [c]
    cmds:
      - rm -rf node-modules

  install:
    desc: Install dependencies
    aliases: [i]
    run: once
    cmds:
      - npm install
    preconditions:
      - test -f package.json
      - test -f package-lock.json
    sources:
      - package.json
      - package-lock.json
    status:
      - test -d node_modules

  run-only:
    aliases: [ro]
    interactive: true
    ignore_error: true
    dotenv: ['.env']
    cmds:
      - task: install
      - task: prisma:generate
      - npx --yes esno src/index.ts

  run:
    desc: Run the project
    aliases: [r]
    cmds:
      - task: docker-compose:up
      - task: prisma:push
      - task: run-only
```

There are a lot of things going on here, so trying to explain some things:

1. The first line is just for auto complete on IDEs (like vscode, zed)
2. We can define global env vars in the script itself (e.g. `DATABASE_URL`)
3. We can [import](https://taskfile.dev/usage/#including-other-taskfiles) part of the scripts. You just need to have your global ones (e.g. terraform, docker-compose, prisma, git, etc). It does work with remote files.
4. You can expose/hide tasks with [internal flag](https://taskfile.dev/usage/#internal-tasks)

## Now the best part - cache

Pay a close look on the install task.

1. it only runs if `package.json` and `package-lock.json` are present ([programatic checks](https://taskfile.dev/usage/#using-programmatic-checks-to-cancel-the-execution-of-a-task-and-its-dependencies))
2. [Prevent unnecessary work](https://taskfile.dev/usage/#prevent-unnecessary-work): it caches the execution using the `source` attribute.
3. at the end it checks if the `node_modules` folder was created.
4. Even caching based on the `source`, if you remove the `node_modules` folder (check) it will run again.

So, you just need to run your daily tasks (e.g. tests, lints, run local) and the script itself will detect what is pending and run it if needed.

No more crazy issues that you forgot to update a dependency or database change.

## Ok, show me a normal day of work

At the first time, you just need to run `task r` to run the app. It will install dependencies, start database, apply db changes and run the app.

But what happen if someone changes a dependency? No problem. Taskfile will detect and will run `npm install` again.

But if someone just changes the db schema? No problem. It will generate the classes again.

If nothing changes, it will just run the app (ok, the docker part is not optimized, but you got the idea).

If need to stop the app and run again you can go direct to `task ro` (ro = run only).

## Full Project

As always, the full sample is available on [github](https://github.com/adamatti/node-sample).

## Closing thoughts

Now go to the [documentation](https://taskfile.dev/usage/).

Remove commands from `README.md` files and confluence/sharepoint. Also remove it from anemic `package.json` 😛

My global taskfiles are shared [here](https://github.com/adamatti/dotfiles/tree/main/taskfile) for reference.

My backup scripts using rsync are also on taskfiles, but this is a subject for another post.


Taskfile - an alternative for Makefiles


Imagine the follow scenario: you do have a node api with typescript, that uses a database (postgres in this sample) and an orm ([prisma](https://www.prisma.io/)). Every time someone in the team adds a new dependency, you need to run `npm install`. 

Database changes? You need to generate the classes again with `npm run prisma generate`. Oh, you also need to apply the changes with `npm run prisma db push` (note: a real world project would use migrations files, so `npm run prisma migrate dev`).

Then you moved to another service. Then moved back... oh, now need to start database again with `docker-compose up -d`. And so one... 

The steps are documented in a readme file. Or you can get it from `package.json` -> `scripts`, but there is no comments there or command order/dependence. 

What do you do? 

#### Makefile to rescue

So, Make is a build automation tool (thanks [wikipedia](https://en.wikipedia.org/wiki/Make_(software)), and it is available by default in all unix systems.

With that you can:
- create goals (tasks)
- create multi line commands
- use vars
- use conditions
- create aliases for the commands
- map task dependencies
- avoid re-running a task if artifact is already created (e.g. build). "Kind" of a cache (strong quotes here)

#### So, show me a sample!

Sure... check this one (full project [here](https://github.com/adamatti/node-sample)):

```makefile
default:
	@echo "Please specify a target to run"

clean:
	rm -rf node_modules

node_modules:
	npm install

docker-compose-up:
	docker-compose up -d

prisma-db-push:
	npx --yes wait-on tcp:5432 && \
	DATABASE_URL=postgres://postgres:sample@localhost:5432/sample npm run prisma db push

prisma-generate:
	npm run prisma generate

run-only:
	DATABASE_URL=postgres://postgres:sample@localhost:5432/sample \
	npx --yes esno src/index.ts

run: docker-compose-up prisma-generate prisma-db-push run-only

format:
	npm run prisma format
	npx --yes biome format --write ./src

# Aliases
install: node_modules
dcu: docker-compose-up
pdp: prisma-db-push
pg: prisma-generate
r: run
ro: run-only
first: clean install run
fresh: first
f: fresh
```

As you can see, a simple task is just 

```makefile
taskname: dependency1 dependency2
    my list of commands
    second command here
    etc
```

with the sample provided, I can install dependencies with `make node_modules`. 

If you run it again, it will do nothing as the `node_modules` folder is already present (task name = folder name).

If I want to run my app the first time, I just need `make fresh`, and it will:
- install dependencies
- start database
- generate orm classes
- update database
- run the app

If I want to run the app again, I don't need the full steps. I can go direct to `make run-only`.

#### Ok, I am lazy than that

With dependencies, you can create aliases. e.g.

```makefile
r: run
ro: run-only
```

Now I just need `make r` to run my app. Or `make ro` to run-only.
Note: I am lazier than that, I also [have an alias](https://github.com/adamatti/dotfiles/blob/bb23c786a3181b977874efe83530b65ec5de804a/zsh/aliases.zsh#L3) to run `make` with `m` only 😛

#### How about the default task?

It uses the first task. Or the one defined on `.DEFAULT_GOAL := help` (add it as one of the first lines).

#### Ok, but how about documentation

There is a [trick](https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html) for that - just add those lines in the beginning of the file:

```makefile
.PHONY: help

help:
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
```

...plus, `## comment` in the tasks you want comments. E.g.

```makefile
clean: ## remove dependencies
	rm -rf node_modules

prisma-generate: ## generate prisma classes
	npm run prisma generate
```

Now you just need to run `make` or `make help` to see all tasks and comments, sorted by task name, e.g.

```shell
> make help
clean                          remove dependencies
docker-compose-up              start dependencies
first                          first execution
format                         format code
fresh                          fresh execution
install                        install dependencies
prisma-db-push                 push prisma schema to database
prisma-generate                generate prisma classes
run-only                       run the project
run                            run the project + dependencies
```

#### Advanced

- How to hide the command from output? Just add `@` in the front of the command.
- call an existing task in the middle/end of my commands? `$(MAKE) your task`
- cache task? the task should match with the output folder or file name

#### Show me the full project

It is [here](https://github.com/adamatti/node-sample), and the makefile is [here](https://github.com/adamatti/node-sample/blob/main/Makefile)

#### Why post about makefile?

Because this is the [most watched video on my channel](https://www.youtube.com/watch?v=h4aV3C6BRDQ). And a lot of developers still adding commands on readme files on partially on package.json. We developers should automate things, even our daily tasks.

Makefiles - How to use them for automations


Just saw this new lib today - [Vinejs](https://vinejs.dev/). It called my attention as it is very similar to [zod](https://zod.dev/), but seems to be faster. Definitely I will consider it for my next pet projects.

#### Basic Example

```typescript
import vine from '@vinejs/vine'

const schema = vine.object({
  email: vine.string().email(),
  password: vine
    .string()
    .minLength(8)
    .maxLength(32)
    .confirmed()
})

const data = getDataToValidate()
await vine.validate({ schema, data })
```

#### Benchmarks

<div style="align:center">
  ![Benchmarks](/assets/posts/2024-04-05-vinejs/benchmark.jpg)
</div>

Memory Leak in nodejs/javascript

Cloudflare workers

Yarn Classic

Taskfile - an alternative for Makefiles

Makefiles - How to use them for automations

Vinejs - js validation lib