Remote Profiling Elixir Over SSH
Erlang comes with an awesome tool called observer, which is readily available to the Elixir user.
If you want to use it, simple run
:observer.start from an IEx session.
An interesting feature is that you can then connect to another node and observer its details. Getting that working when the other node is on a different machine accessible only via SSH is another story. This is that story!
A brief trip into the world of SSH
SSH is something many developers use on a daily basis. The way we normally use it is to provide us with a secure connection to a remote server, allowing us to run commands as if we were in the same room.
Another thing we can to is to forward local ports to remote machines. Let me explain with an example.
ssh -N -L 9001:localhost:9001 email@example.com
If we now do anything that involves connection to port 9001, it will be sent to the remote machine. This gets kind of cool and allows you to use local instance of utilities like
psql, but connected to a remote machine! This means your interactions are as fast as local, you don't get that SSH typing delay that (for me at least) causes me to make lots of typos because I'm second guessing myself!
The -N flag means we don't want to interact directly with the session, so it will just sit there and wait.
This port forwarding becomes important because, although Erlang / Elixir can happily communicate with other instances over a local network, it is not happy to do so over SSH.
Let's fix that.
Starting IEx with some restrictions
By default the Erlang VM will use a bunch of different ports for distributed communication. We're going to force its hand, enabling us to guarantee which ports we should forward.
On your remote machine, run
Addendum - this will only work if you have a named node running
$ epmd -names epmd: up and running on port 4369 with data:
You should see something similar to the output above.
EPMD is the Erlang Port Mapper Daemon. It is kind of like a DNS server for Erlang nodes.
On our remote machine we can start IEx with a specific port on which to listen, rather than a range of ports.
In this example I wanted to profile a Phoenix application.
iex --name firstname.lastname@example.org --cookie 123 --erl "-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001" -S mix phoenix.server
Let's break down what's been done here.
--name gives our node a name, in this instance I'm explicitly stating it's on localhost. The alternative
--sname didn't seem to play nice, but that may be operator error!
--cookie Is where we set a shared secret that gives nodes permission to talk to each other. In this instance I didn't care about making it a difficult cookie as I'm on a local network with non-production systems.
--erl Allows us to pass options to the Erlang VM itself.
-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001 Is essentially saying, to the VM, you may only listen on port 9001.
Setting up as SSH tunnel
We now know that ports 4369 and 9001 need to be linked.
ssh -N -L 9001:localhost:9001 -L 4369:localhost:4369 email@example.com
Here we're linking two ports at the same time to the remote machine.
You may be prompted for a password at this point, depending on whether you're set up key authentication or not.
Seeing all the nodes
In another local terminal session, run
epmd -names, you should see something like this.
epmd: up and running on port 4369 with data: name bench at port 9001
The fact that it shows
name bench at port 9001 tells us that we now have the ability to connect to that node.
Connecting the Erlang observer
We'll now fire up and Erlang Shell running the observer.
erl -name firstname.lastname@example.org -setcookie 123 -run observer
Note that the syntax is slightly different, but we give it a name different to the remote node, but the same cookie. This will now start the Erlang observer.
Once this is running, click on the Nodes option on the menu bar and select
Connect node, enter the name we used earlier
If all is well, you will now see your remote application's profiling data. Note that in this example, I've connect to a node running on a Raspberry PI, so we've got 4 cores on an ARM processor.
I'll just throw a bit of load at it ... and
Being able to communicate with a running node, remotely, over SSH gives us the ability to do lots of interesting experiments. Load testing, for example. The really nice thing about this is that it's much closer to real world usage than running benchmarks locally.
In an upcoming post I'll share what I've learned from running Phoenix on a Raspberry Pi.