TL;DR.
This post is a guide to installing, and using, LiteLLM. It is a tool for accessing a large language model (LLM), like Llama 2 and Mistral, using an IP address and port number. I also cover Miniconda, Ollama, and Llama2. I share my experience with creating virtual environments, managing Python projects, by demonstrating how LiteLLM is used to access to LLMs across a local network, showing the potential of using this tool in AI projects and applications.
Attributions:
https://docs.anaconda.com/free/miniconda/index.html↗,
https://ollama.com↗.
https://litellm.ai/↗.
https://solodev.app/installing-ollama, and
https://solodev.app/installing-miniconda.
An Introduction.
LiteLLM is a tool for making LLMs accessible across networks.
The purpose of this post is to show how to make LLMs accessible.
The Big Picture.
LLMs are not usually reachable using IP addresses and port numbers. However, such a feature would make LLM accessibility more convenient. Thankfully, LiteLLM is able to act as a proxy so that LLMs are usable across my LAN.
Prerequisites.
Updating my Base System.
- In the (base) terminal, I update my base system:
sudo apt clean && \
sudo apt update && \
sudo apt dist-upgrade -y && \
sudo apt --fix-broken install && \
sudo apt autoclean && \
sudo apt autoremove -y
What is Ollama?
Ollama is a tool for downloading, setting up, and running LLMs (large language models). It helps me access powerful models like Llama 2 and Mistral and lets me run them on my local Linux, macOS, and Windows systems.
https://ollama.com/↗.
Ollama must be installed (ollama -v) before continuing with this post.
Running the Llama2 Model.
- In the (base) terminal, I change to the Ollama environment:
conda activate Ollama
NOTE: Install Ollama to successfully run this command.
ollama run llama2
What is Anaconda and Miniconda?
Python projects can run in virtual environments. These isolated spaces are used to manage project dependencies. Different versions of the same package can run in different environments while avoiding version conflicts.
venv is a built-in Python 3.3+ module that runs virtual environments. Anaconda is a Python and R distribution for scientific computing that includes the conda package manager. Miniconda is a small, free, bootstrap version of Anaconda that also includes the conda package manager, Python, and other packages that are required or useful (like pip and zlib).
http://www.anaconda.com/↗,
https://docs.anaconda.com/free/miniconda/index.html↗, and
https://solodev.app/installing-miniconda.
Miniconda must be installed (conda -V) before continuing with this post.
Creating a Miniconda Environment.
- In the second (base) environment, I use the
conda command to display a list of Miniconda environments:
conda env list
- I use
conda to create, and activate, a new environment named (-n) (LiteLLM):
conda create -n LiteLLM python=3.11 -y && conda activate LiteLLM
NOTE: This command creates the (LiteLLM) environment, then activates the (LiteLLM) environment.
Creating the LiteLLM Home Directory.
NOTE: I will now define the home directory in the environment directory.
- I create the LiteLLM home directory:
mkdir ~/LiteLLM
- I make new directories within the (LiteLLM) environment:
mkdir -p ~/miniconda3/envs/LiteLLM/etc/conda/activate.d
- I use the Nano text editor to create the
set_working_directory.sh shell script:
sudo nano ~/miniconda3/envs/LiteLLM/etc/conda/activate.d/set_working_directory.sh
- I copy the following, paste (CTRL + SHIFT + V) it to the
set_working_directory.sh script, save (CTRL + S) the changes, and exit (CTRL + X) Nano:
cd ~/LiteLLM
- I activate the (base) environment:
conda activate
- I activate the (LiteLLM) environment:
conda activate LiteLLM
NOTE: I should now, by default, be in the ~/LiteLLM home directory.
What is LiteLLM?
LiteLLM provides LLMs with IP addresses. It is a unified interface that calls 100+ LLMs using the same Input/Output format, supporting OpenAI, Huggingface, Anthropic, vLLM, Cohere, and even custom LLM API services.
https://litellm.ai/↗.
Installing, and Using, LiteLLM.
pip install litellm 'litellm[proxy]'
- I use LiteLLM to run Llama2:
litellm --port 6000 --model ollama/llama2
NOTE: The Llama2 model is now accessible from http://localhost:6000.
The Results.
LiteLLM helps me set up environments that harness the power of large language models (LLMs). This tool provides a unified interface for accessing many LLMs. I also used Miniconda to manage my Python environments and Ollama to download, and run, LLMs like Llama 2. This exploration used virtual environments to manage project dependencies but also highlighted AI tools that make LLM technologies more accessible. My successful setup, and use, of LiteLLM shows how LLMs can be easily integrated into various AI projects and applications.
In Conclusion.
I dived into setting up LiteLLM. It is used to access an LLM, like Llama 2 and Mistral, using an IP address and port number. LiteLLM is a unified interface that simplifies the integration of many LLMs, making them accessible across my LAN. I also used tools like Miniconda and Ollama.
The results show that Miniconda, Ollama, and LiteLLM make LLM technologies more accessible, and also shows the potential of using these tools in LLM projects. This was a deep dive into the future of AI technologies and their integration into my projects and applications.
What's your experience with LLMs? Have you used tools like Miniconda, Ollama, and/or LiteLLM in your projects? Let's share insights and learn from each other's experiences in harnessing the power of large language models.
Until next time: Be safe, be kind, be awesome.
#LiteLLM #Miniconda #Ollama #LLMs #Python #ArtificialIntelligence #AIModels #TechTools #VirtualEnvironments #Programming
