Senior Staff Applied ML Engineer - Consumer Agents

Senior Staff Applied Machine Learning Engineer, Consumer Agents

Location: Remote — North America

About the role

The Consumer Agent team builds Shopify's consumer-facing agentic shopping experiences — the AI agents inside the Shop app and on merchant storefronts that help millions of buyers search, discover, and buy. As a Senior Staff Applied Machine Learning Engineer, you'll be the person we point at the hardest problems on the surface — distillation to own-GPU inference, agentic memory, whole-page optimization, propensity modeling, or the evaluation rigor that ties them together — and trust to drive it to a shipped, measured outcome. This is a senior individual-contributor seat with deep autonomy: you'll own ambiguous, high-leverage problems end to end, raise the bar for evaluation across the team, and ship weekly on a fast-moving consumer surface. You'll work at the intersection of LLM orchestration, model training and distillation, search and retrieval, and cost/latency engineering, partnering with frontier model labs and ML teams across Shopify.

Key Responsibilities

• Build and improve student models through supervised fine-tuning and knowledge distillation, including the serving, generation-termination, and throughput-economics work that own-GPU inference demands.
• Design and raise the bar for ML evaluation across the team — judge design, offline and online metrics, and the judgment to know when a number is real.
• Develop agentic memory and personalization signals — buyer and user representations, profile generation — that feed predictive layers.
• Work on whole-page optimization: page composition, ranking, and blending for the agent results surface.
• Build propensity and predictive models for intent and purchase propensity across agent surfaces.
• Move fluidly between distillation, memory, ranking, propensity, and evaluation as priorities shift, without lengthy re-tooling.
• Apply rigorous experimentation on a high-traffic surface — proper holdouts, A/B rigor, and counterfactual or off-policy evaluation where online tests aren't enough.
• Integrate cleanly with an LLM orchestration stack already in flight — tool calling, structured output, and retrieval-augmented patterns.
• Become the technical reference point the team routes its hardest model and evaluation calls to.
• Contribute to cross-team ML architecture conversations across Shopify's ML organization and external model-lab partnerships.
• Ship weekly, absorbing ambiguity rather than adding coordination overhead.

Qualifications

• You've shipped LLM systems to production at real scale — systems with real users, not research prototypes.
• Hands-on depth in model training and distillation: supervised fine-tuning and knowledge distillation in practice.
• Comfort with the serving and throughput-economics side of inference, not just the training run.
• Rigorous evaluation methodology: judge design, offline and online metrics, and the judgment to distinguish a real result from an artifact.
• You operate as a senior individual contributor with high autonomy — you take an ambiguous, hard problem and drive it quickly to a measured outcome without hand-holding.
• Breadth over a single specialty: you can move across distillation, memory, whole-page optimization, propensity, and evaluation.
• Strong communication and collaboration in a fast-paced, cross-functional environment.
• Able to work with significant overlap with North America time zones.

Nice to have

• E-commerce, marketplace, commerce-search, recsys, or ads background — with instincts for selection bias, position bias, and the gap between offline and online metrics.
• Inference and serving internals: vLLM or similar engines, structured and constrained decoding, KV-cache and GPU throughput economics, and latency budgets.
• Reinforcement learning experience — RLHF, RLAIF, or preference optimization.
• Propensity, ranking, or classical applied ML at scale: gradient boosting, neural ranking, calibration, position-bias correction.
• Counterfactual evaluation experience — inverse propensity scoring, doubly-robust estimators, off-policy evaluation.
• Experience working alongside or with frontier AI products and agentic systems.