LLM API Performance Evaluation Tool Guide
This article provides a quick start guide for Linux and Windows platforms, including command examples for downloading, configuring, and running the tool.
Re-evaluating: The True Power of Flash Attention 2
This article compares the performance differences of vllm when using xformers and flash attention 2 as the backend attention mechanism.
Running Large Language Models on a VPS
This post documents how to run LLMs on a 1C1G VPS using Ollama.
Large Language Model Inference Framework Throughput Comparison: VLLM | SGLang | LMDeploy
This article compares the throughput of three large language model inference engines, VLLM, SGLang, and LMDeploy, in a short-input, long-output scenario. The unit of measurement is output tokens per second.
How to Make Your Website More Secure? How to Restrict Services Started via Docker Containers with UFW? UFW One-Click Script
This article explains how to configure the UFW firewall using a one-click script to restrict network access for Docker container services, enhancing website security.