
There was a requirement here to stream data from [mongodb](https://www.mongodb.com/) to [Elasticsearch](https://www.elastic.co/), and I decided to research the existing tools.

We could use [elasticsearch-river-mongodb](https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki), [implement a stream process in node](https://medium.com/@amulyakashyap09/stream-mongodb-to-elasticsearch-using-node-js-755b407eb744), some payed solutions... then I found [monstache](https://rwynn.github.io/monstache-site/) - a stream implementation in golang.

## The good parts

- Monstache is just a single binary without dependencies on runtimes like Ruby, Python or PHP - a [scratch image](https://hub.docker.com/_/scratch) should do the work
- It does accept [transformations](https://rwynn.github.io/monstache-site/advanced/#transformation) - working sample [here](https://github.com/adamatti/learn-monstache/commit/51ab57c1be5adb3b60555a74838e49c50ffa05d4)
- [Mappings](https://rwynn.github.io/monstache-site/advanced/#index-mapping) - working sample [here](https://github.com/adamatti/learn-monstache/commit/5ff11d50148354a28c8e027c7935fa0129607013)
- [Versioning](https://rwynn.github.io/monstache-site/advanced/#indexing-metadata)? - working sample [here](https://github.com/adamatti/learn-monstache/commit/9768a64a0e12ac4c0926049b6731c377013df9bc)
- Fetch references? check [embedding documents](https://rwynn.github.io/monstache-site/advanced/#embedding-documents)
- A lot of mongo/elastic [configurations](https://rwynn.github.io/monstache-site/config) - e.g. replay, resume, resume-strategy
- The [docker image](https://hub.docker.com/layers/rwynn/monstache/6.7.17/images/sha256-a275e45f20e2f2e64f30140479469b1241be340cd81ac5641ac391edcb1d092f?context=explore) is less than 15 mb (compressed size)

## Ok, how can I make it faster?

From [the docs](https://rwynn.github.io/monstache-site/advanced/#middleware):

> It is HIGHLY recommended to use a golang plugin in production over a javascript plugin due to performance differences. Currently, golang plugins are orders of magnitude faster than javascript plugins. This is due to concurrency and the need to perform locking on the javascript environment. Javascript plugins are very useful for quickly prototyping a solution, however at some point it is recommended to convert them to golang plugins

For a golang working sample, check [my repo](https://github.com/adamatti/learn-monstache/commit/ae4d813485246f284d4583c4712a4adfaa94d44c)

## The bad parts

It uses [streams](https://x.com/baptistejamin/status/1615310358528376832), and it seems it is not performant as using oplog ([ref](https://jira.mongodb.org/browse/SERVER-46979))

## Conclusion

I would give it a try and perform additional performance tests prior to go to prod.

It is nice it does support [multiple workers](https://rwynn.github.io/monstache-site/advanced/#workers), [High Availability](https://rwynn.github.io/monstache-site/advanced/#high-availability), etc

It has almost 9yrs since [first commit](https://github.com/rwynn/monstache/commit/04c590bf0896330405eeb12614aba1fc8e570fba), [1.3k starts on github](https://github.com/rwynn/monstache), [videos on youtube](https://www.youtube.com/results?search_query=monstache+elastic)

## References

- [Monstache Github Repo](https://github.com/rwynn/monstache)
- [Monstache Website/Documentation](https://rwynn.github.io/monstache-site/)
- [Github - my working sample](https://github.com/adamatti/learn-monstache)
- [A docker compose with all you need](https://github.com/rwynn/monstache-showcase/blob/master/docker-compose.sc.yml)
- [Step by Step how to create a golang plugin](https://github.com/rwynn/monstache/wiki/Go-plugin-guide)


A tool to stream data from mongodb to elasticsearch

Monstache


Ok, Make/Makefiles are great. I spoke about then in the [last post](/blog/posts/2024-04-08-makefile).

But what if:

- I want to break them in different files
- Use a better syntax (maybe yaml?)
- Cache the tasks, even the ones that don't generate artifacts (e.g. lint)
- put tasks and aliases close
- validate the task execution
- have global tasks to run in any folder (e.g. backup)
- dry run
- watch task

#### Taskfile to rescue

Just need to [install Task](https://taskfile.dev/installation/) (or go-task on [github](https://github.com/go-task/task)).

The [documentation](https://taskfile.dev/) is pretty simple IMHO, but I want to highlight some things that I do use in daily bases that I think it does a great difference.

## Show me the code

For this I will use the sample from [previous post](/blog/posts/2024-04-08-makefile) ([makefile here](https://github.com/adamatti/node-sample/blob/main/Makefile))

```yaml
# yaml-language-server: $schema=https://json.schemastore.org/taskfile.json
---
version: '3'

env:
  DATABASE_URL: postgres://postgres:sample@localhost:5432/sample

includes:
  docker-compose:
    aliases: [dc]
    taskfile: ./Taskfile-docker-compose.yaml
  prisma:
    aliases: [p]
    taskfile: ./Taskfile-prisma.yaml

tasks:
  default:
    desc: Show help
    aliases: [h]
    silent: true
    cmds:
      - task -l --sort alphanumeric

  clean:
    desc: Remove node modules
    aliases: [c]
    cmds:
      - rm -rf node-modules

  install:
    desc: Install dependencies
    aliases: [i]
    run: once
    cmds:
      - npm install
    preconditions:
      - test -f package.json
      - test -f package-lock.json
    sources:
      - package.json
      - package-lock.json
    status:
      - test -d node_modules

  run-only:
    aliases: [ro]
    interactive: true
    ignore_error: true
    dotenv: ['.env']
    cmds:
      - task: install
      - task: prisma:generate
      - npx --yes esno src/index.ts

  run:
    desc: Run the project
    aliases: [r]
    cmds:
      - task: docker-compose:up
      - task: prisma:push
      - task: run-only
```

There are a lot of things going on here, so trying to explain some things:

1. The first line is just for auto complete on IDEs (like vscode, zed)
2. We can define global env vars in the script itself (e.g. `DATABASE_URL`)
3. We can [import](https://taskfile.dev/usage/#including-other-taskfiles) part of the scripts. You just need to have your global ones (e.g. terraform, docker-compose, prisma, git, etc). It does work with remote files.
4. You can expose/hide tasks with [internal flag](https://taskfile.dev/usage/#internal-tasks)

## Now the best part - cache

Pay a close look on the install task.

1. it only runs if `package.json` and `package-lock.json` are present ([programatic checks](https://taskfile.dev/usage/#using-programmatic-checks-to-cancel-the-execution-of-a-task-and-its-dependencies))
2. [Prevent unnecessary work](https://taskfile.dev/usage/#prevent-unnecessary-work): it caches the execution using the `source` attribute.
3. at the end it checks if the `node_modules` folder was created.
4. Even caching based on the `source`, if you remove the `node_modules` folder (check) it will run again.

So, you just need to run your daily tasks (e.g. tests, lints, run local) and the script itself will detect what is pending and run it if needed.

No more crazy issues that you forgot to update a dependency or database change.

## Ok, show me a normal day of work

At the first time, you just need to run `task r` to run the app. It will install dependencies, start database, apply db changes and run the app.

But what happen if someone changes a dependency? No problem. Taskfile will detect and will run `npm install` again.

But if someone just changes the db schema? No problem. It will generate the classes again.

If nothing changes, it will just run the app (ok, the docker part is not optimized, but you got the idea).

If need to stop the app and run again you can go direct to `task ro` (ro = run only).

## Full Project

As always, the full sample is available on [github](https://github.com/adamatti/node-sample).

## Closing thoughts

Now go to the [documentation](https://taskfile.dev/usage/).

Remove commands from `README.md` files and confluence/sharepoint. Also remove it from anemic `package.json` 😛

My global taskfiles are shared [here](https://github.com/adamatti/dotfiles/tree/main/taskfile) for reference.

My backup scripts using rsync are also on taskfiles, but this is a subject for another post.


Monstache

Taskfile - an alternative for Makefiles