Set up MLFlow

Set up MLFlow

Download Anaconda how you normally would from https://www.anaconda.com/download

Once downloaded, run this command to make the .sh file executable, and then execute it as your normal user, not as root.

chmod +x <file_name_here>.sh
./<file_name_here>.sh

It will be installed to /home/<your_user>/anaconda3

Once it is installed, add Anaconda to your path

Anaconda 2 :
export PATH=~/anaconda2/bin:$PATH

Anaconda 3 :
export PATH=~/anaconda3/bin:$PATH

Create a new environment in Anaconda

Create a new environment by running this command

conda create -n myenv anaconda

Or create an environment by specifying a python version

conda create -n myenv python=3.9 anaconda

Activate your conda environment

conda activate myenv

You might get an error saying you need to run “conda init” first. If you do run this command

conda init

Then close your terminal, and open a new one. You will now see “(base)” in your terminal, indicating you are in the default anaconda environment

Run this command to list your conda environments:

conda env list

Activate your conda environment again for the rest of the steps.

Access Jupyter Server over the network with a password

This approach adds all the config to the “server” configuration (the new way of doing things), and we use the “server” command and config files instead of the “notebook” command and config files.

Create a Jupyter Server config file

jupyter server --generate-config

The config file will be written to /home/<your_user>/.jupyter/jupyter_server_config.py\

Add the following to the bottom of the config file

This will make Jupyter Notebook accessible over the network so it can be accessed from anywhere

c.ServerApp.ip = '0.0.0.0'
c.ServerApp.port = 8888
c.ServerApp.password_required = True
c.ServerApp.open_browser = False

Save and exit

Set a password for jupyter server

Using “notebook” instead of “server” in the below command does not seem to affect anything because even when using “notebook” the config goes to /home/<your_user>/.jupyter/jupyter_server_config.json

jupyter server password

To start Jupyter, you will now run this command

jupyter server

Keep the terminal open to keep the Jupyter Server alive.

Update your default Jupyter Server notebook_dir location

If you want to see all the current config and data directories for Jupyter, run this command

/srv/jupyter

If you want to update the default location where notebook files are stored, add this to the end of /home/<your_user>/.jupyter/jupyter_server_config.py

c.NotebookApp.notebook_dir = '<your new location here>'

Install Conda Navigator

Make sure you are in your base conda environment. If you are still in your virtual environment run

conda deactivate

Your terminal should show (base). If not, run

conda activate

Now run

conda install -c anaconda anaconda-navigator

And then launch it with

anaconda-navigator

To update all anaconda packages run

conda update --all

This will only update the packages in your active environment

Install MLFLow from Conda Forge

Look for a package in Anaconda Forge.

conda search -c conda-forge mlflow

Then install it

conda install -c conda-forge mlflow

Start MFLow Web UI

Run this command in a terminal

mlflow server --host 0.0.0.0 --port 8080

0.0.0.0 ensures it is available on your network, not just localhost.

Quick MLFlow overview

Getting started: https://mlflow.org/docs/latest/getting-started/logging-first-model/step1-tracking-server.html

This tutorial will explain how to create the following:

Experiments, which are essentially the base of everything. Every training step gets logged under an experiment so that you can visualize the experiment’s loss, accuracy, etc

When you click on an experiment it shows you all the properties of that experiment, such as the parameters, artifacts (pip requirements, input examples, etc) for the model, etc

Here all the artifacts for this experiment are displayed:

When you are satisfied with an experiment, i.e. there is a specific run you like, you can register a model from it.

This is how you can register a model:

Click on the experiment and top right click on “Register Model”

This is where you can find your registered models:

When you click on your model you can see the various versions. A model can have multiple versions and using MLFlow you can pull a specific model and version):

You can then click on a version and “promote “it to e.g. production. Promotion is a maturity thing and the link below talks more about it.

This shows how to pull models from the registry and talks a lot more about promoting models: https://mlflow.org/docs/latest/model-registry.html

Here is a quick snippet from this URL showing how to use one of your registered models: