Press Kit

Longer

ML Shortcuts turns a machine-learning model into a way to control a Mac. Users import a Core ML model, for example one they trained in Create ML, and the app lists its classes. Next to each class they press the keyboard shortcut it should trigger. When the model recognises a class, a thumbs up or a clap for example, ML Shortcuts presses those keys.

It works with five kinds of models: sound classifiers on the microphone, and image classifiers, hand pose classifiers, hand action classifiers and object detectors on the camera. Hand pose and object detection models can also move the pointer, and hand signs can click and drag. A confidence threshold and an option to ignore repeated detections keep it calm in a real room.

The app lives in the menu bar and runs the model entirely on the Mac, without an account or an internet connection. Designers use it to try gesture interfaces in Figma prototypes without code; everyone else can switch Spaces, save or lock the screen with a gesture. A sample hand action model comes with the download.