A paper posted to arXiv's information retrieval section describes RecoGEM, a serving system for generative recommendation, an approach in which a model generates item identifiers step by step instead of scoring a list of candidate items. The authors say the approach promises higher recommendation quality but makes serving much harder, and that neither conventional recommender pipelines nor existing LLM serving stacks fit it well.
RecoGEM splits the work between a dedicated runtime, which handles tasks such as candidate-state transitions and live artifact updates, and an execution engine tuned for the short token sequences and large candidate pools typical of generative recommendation.
According to the authors, the system has been deployed for over nine months across their production lines, serving applications with hundreds of millions of users and supporting more than ten model families. On their OneModel production workloads, they report over 1,000 queries per second per serving instance, P99 latency below 35 ms and 2.88x the throughput of representative baselines. The abstract does not name the company that runs it.