microsoft/sarathi-serve

A low-latency & high-throughput serving engine for LLMs

↗ Open on GitHub

Stars

520

Forks

66

Language

Python

License

Apache-2.0

Last push

8 months ago

Python llm-inference transformer pytorch llama

No related intel yet

This repo has not appeared in any of the sources the radar tracks. The collector runs on a schedule — check back once it covers this repo.