For detailed instructions and more information, launch UI Browser and choose Help > UI Browser Help.
UI Browser and GUI Scripting
GUI Scripting is a technology for Mac OS X that adds simple, plain-English commands to AppleScript to control applications by scripting their user interface. Because it is based on Apple's Accessibility API built into every standard User Interface element in Cocoa and Carbon applications, it allows scripts to control applications that are not otherwise scriptable, bringing previously impossible automation tasks in reach of every scripter.
But there is one aspect of GUI Scripting that makes every script difficult: how to figure out the containment hierarchy of the target application's UI elements. Every application has its own idiosyncratic arrangement of containers, many of which are internal coding conveniences and not visible to users. Furthermore, many UI elements have no names and therefore have to be scripted by their index numbers—which often have no relationship to their visual layout. While you can use AppleScript itself to decipher these mysteries, it is very awkward and time consuming.
UI Browser saves the day! Just fire it up, choose the target application, or use the built-in Screen Reader or press a hot key to read the screen under the mouse, and look at the browser view in the main window. A few clicks and a couple of scrolls, and you see the exact containment hierarchy, complete with every UI element's title (if it has one) and its AppleScript index number.
We even save you the trouble of writing it down. With UI Browser's unique AppleScript menu, several different kinds of AppleScript statement are generated for you, including object references, get and set commands for object properties, action commands, and keystroke commands. The generated AppleScript statement appears in a separate window. If you aren't sure whether you prefer the AppleScript Name reference form or the Index reference form when both are available, click a button to switch instantly between reference forms! The generated AppleScript statement is already selected for you, ready to edit, or to copy and paste or drag into your favorite script editor. Better yet, set a preference to copy it to the clipboard or send it directly to your default script editor automatically.
It doesn't end there. Before you design your script, and while you're fine tuning and testing it, you can use UI Browser's ability to set attributes and send actions to make sure the target application will respond as you expect. Not every application is fully accessible, in part because some applications include custom user controls that haven't yet been made accessible. Instead of banging your head against the wall for hours because you think you haven't written your script correctly, just try to set the property or send the action using UI Browser. If it doesn't work in UI Browser, it most likely isn't your script that is at fault. Or, just maybe, you will find a way to make the target application respond that wasn't immediately obvious to your AppleScript instincts. UI Browser is not based on AppleScript, System Events, or GUI Scripting; it is based directly on the same Accessibility API that GUI Scripting uses. If UI Browser can't make the target dance, GUI Scripting probably can't either.
UI Browser is a user interface navigator
With UI Browser, you can "see" almost every window, view, control, and menu item of a target application, without bringing the target to the front. You can see at a glance, in a familiar Mac OS X browser view, where all of them are in the containment hierarchy, easily navigating through windows, sheets, drawers, tab views, and other containers. To keep you from getting lost, UI Browser's Path view gives you a compact outline of the path from the target application's root element to any UI element you select, with an ordered list of all intermediate elements, including many that are not normally visible to users. UI Browser can even highlight the selected UI element on the screen to help you identify it. You can also see in UI Browser's Attributes view dozens of properties of any UI Element in the target, including its role, title, position, and size, and its current state.
UI Browser is a user interface actor
Using standard controls suitable for each type of data the target application recognizes, you can set the value of every user-settable attribute in the target—the size and position of windows, whether the application is frontmost or hidden, whether a window is focused or minimized, whether an outline or table row is selected, and much more, all without touching the target application's GUI. You can also perform every action supported by the target, such as clicking or pressing its checkboxes and buttons, showing menus, and confirming text field entries. You can even send keyboard shortcuts to the target application and "type" individual characters into the active text field or text view of the target application.
UI Browser is a user interface observer
You can tell UI Browser to observe notifications that are broadcast by Mac OS X applications when any changes take place in their user interface—whether because a user clicked a control in the target application, chose a menu item, or typed some characters, or because an AppleScript command took effect or a network administrator or user took some action, or because you did something to the target with UI Browser itself! You can use notifications to monitor an application's response to GUI Scripting and other commands.
UI Browser and Accessibility
PFiddlesoft has unmatched experience controlling the graphical user interface of applications running on Apple computers using AppleScript and other technologies. Our products carry on the tradition of PreFab Player, a seminal product released in 1994 for the classic Mac OS. Now, PFiddlesoft brings our expertise to you on Mac OS X with UI Browser, a utility to support Apple's GUI Scripting and Accessibility technologies.
Apple released Mac OS X 10.3 Panther in October 2003, including by default full support for GUI Scripting. GUI Scripting is a technology for Mac OS X having many similarities to PreFab Player. It allows you to manipulate any Macintosh application by writing simple AppleScript scripts to control its menus, buttons, and other controls, even if the target application is not scriptable. For more information about GUI Scripting, visit the Mac OS X Automation website.
Originally, GUI Scripting was made available in Mac OS X 10.2.3 Jaguar through a public beta version of Apple's System Events application. Because GUI Scripting has been installed by default on every Macintosh computer since Mac OS X 10.3 Panther, you can distribute your scripts for use "out of the box" without requiring any special supporting software.
GUI Scripting brings Apple's Accessibility technology to every scripter, enabling you to explore and manipulate the user interface of almost all Mac OS X applications running on your computer. It doesn't matter whether they were written for the Cocoa environment or the Carbon environment, or whether they were written in C, C++, or Objective-C—if they were written using one of Apple's standard programming environments, Apple has already made sure they are accessible.
PreFab UI Browser and Apple's GUI Scripting both rely on Accessibility technology that is designed to make the Macintosh more accessible to persons with disabilities. This technology was new to the Macintosh in Mac OS X 10.2 Jaguar, and Apple has made substantial changes since then. As Apple continues to improve Accessibility in the future, UI Browser will automatically take advantage of many of these changes.
Note that custom controls in some applications may not be accessible to GUI Scripting or UI Browser until their developers release a new version that supports accessibility.
This page was first published by PFiddlesoft on May 25, 2010.
Copyright © 2003-2010 Bill Cheeseman. Used by permission. All Rights Reserved.
PFiddlesoft, PFiddle Software, pfiddle, pfiddles, the PFiddlesoft logo, Wheel of Access, and Applidude are trademarks of PreForm Assistive Technologies, LLC.