UI Browser icon

UI Browser — Support

The ultimate assistant for Accessibility and GUI Scripting

Scroll ↕

For detailed instructions and more information about using UI Browser, launch UI Browser and choose Help > UI Browser Help. If you have questions that aren't answered in the Help book or on this website, send email to us at support@pfiddlesoft.com.

Join the UI Browser Users Group on Yahoo Groups to participate in ongoing discussion of GUI Scripting on macOS using the UI Browser application.

Resources for GUI Scripting

Consult these additional sources of information about GUI scripting:

  • See the "Use GUI Scripting" section of the "Get Started With AppleScript" chapter in "Take Control of Automating Your Mac" by Joe Kissel (Second Edition, Take Control Books, 2017). Two older books about AppleScript include much more in-depth discussions of GUI Scripting, both of them still useful today: Lesson 28 in "Apple Training Series: AppleScript 1-2-3" by Sal Soghoian and Bill Cheeseman (Peachpit Press, 2009), and Chapter 24 in "AppleScript: The Definitive Guide" by Matt Neuburg (Second Edition, O'Reilly, 2006).
  • Apple provides official documentation of GUI Scripting and example scripts in the "Automating the User Interface" section of the "Performing Common Scripting Tasks" chapter in Apple's Mac Automation Scripting Guide.
  • For a complete GUI Scripting syntax reference, open the terminology dictionary of Apple's System Events application in your favorite script editor. System Events is located at /System/Library/CoreServices/System Events.app. GUI Scripting terminology is defined in the Processes Suite in the System Events dictionary.
  • Several useful and informative sample GUI Scripting scripts are included on the UI Browser distribution disk image.
  • Apple provides additional sample GUI Scripting scripts. You can find them on your computer in the Local Scripts folder at /Library/Scripts/UI Element Scripts or, if Apple's Script menu extra is in your menu bar, by choosing Open Scripts Folder > Open Computer Scripts Folder > UI Element Scripts.
  • A brief description of GUI Scripting and a few sample scripts are available at Graphic User Interface (GUI) Scripting, part of The macOS Automation Sites managed by Sal Soghoian, Apple's former product manager for AppleScript and automation technologies. At this writing, the downloadable scripts are mostly obsolete but demonstrate how to script System Preferences settings that are not scriptable on their own.
  • Don't forget to read the UI Browser Help book! There's a wealth of information available under UI Browser's Help menu.
  • Read the GUI Scripting Issues and Tips sections, below.

If you aren't sure that GUI Scripting will meet your needs, take advantage of UI Browser's free 30-day trial period to verify that UI Browser and GUI Scripting will work with your target applications.

GUI Scripting Issues

AppleScript script statements generated using UI Browser's AppleScript menu sometimes fail to work even though UI Browser generates valid GUI Scripting statements using correct syntax as defined by Apple. There are two main reasons for this:

  • Some applications do not fully or correctly implement Apple's Accessibility API, on which GUI Scripting depends. A number of well-established macOS applications were originally written using older toolbox code for buttons, menus, and other UI elements. As a result, they could be unresponsive to UI Browser and GUI Scripting. As time has gone by, most of these applications have been updated to incorporate more modern Cocoa UI element code, which incorporates full Accessibility support, or to add custom Accessibility support of their own. If you encounter an application that does not appear to work well with UI Browser or the scripts that it generates, this may be the reason.
  • The Accessibility API sometimes doesn't work perfectly. For a very old example, in Mac OS X 10.3 (Panther), the statement 'get attribute "AXParent" of scroll area 1 of window 1' returned no result, when it should have returned the UI element object representing window 1. Mac OS X 10.4 (Tiger) resolved this and all similar UI element navigation issues. Similarly, in Mac OS X 10.5 (Leopard) or earlier, text inserted into a text area or text field often failed to "stick", disappearing when the user clicked into another text area or text field. Mac OS X 10.6 (Snow Leopard) resolved this issue.

GUI Scripting Tips

These are general tips on GUI Scripting, not necessarily related to UI Browser.

1. Where can I find sample scripts for GUI Scripting?

UI Browser comes with a collection of advanced sample scripts that showcase GUI Scripting features and techniques. These include, for example, scripts showing how to script the menu items in Dock items and in menu extras (menu extras are the menus at the right end of the menu bar). These UI Browser sample scripts are included on the UI Browser distribution disk image, and you may drag them into the UI Element Scripts (PFiddlesoft) folder in the Local Scripts folder at /Library/Scripts/UI Element Scripts (PFiddlesoft). You may find several example scripts from Apple in the same location, at /Library/Scripts/UI Element Scripts, installed as a standard feature of macOS. Apple also provides example scripts in the "Automating the User Interface" section of the "Performing Common Scripting Tasks" chapter in Apple's Mac Automation Scripting Guide. Several sample scripts are available online at the The macOS Automation Sites' Graphic User Interface (GUI) Scripting page. Finally, read Lesson 28 in "Apple Training Series: AppleScript 1-2-3" by Sal Soghoian and Bill Cheeseman (Peachpit Press, 2009), and Chapter 24 in "AppleScript: The Definitive Guide" by Matt Neuburg (Second Edition, O'Reilly, 2006).

2. Why don't Apple's sample scripts in the UI Element Scripts folder work in Mojave?

Apple provides several sample GUI Scripting scripts on your computer at /Library/Scripts/UI Element Scripts, as noted above. However, some of them were written for an earlier version of macOS, and they don't work correctly in newer versions, including macOS Mojave 10.14. This illustrates a well-known feature of GUI Scripting: it is acutely dependent on the user interface of the target application, which may change from version to version of the target application.

Here are simple instructions for revising the sample scripts in question so that they will run correctly on macOS Mojave 10.14:

• The “Get User Name.applescript” UI element script fails with an error alert.

The script should display a dialog saying “Full name: ”, but instead it fails with this error message: ‘Error: System Events got an error: Can’t get text field "Full name:" of tab group 1 of window “Users & Groups” of process “System Preferences”.’

To fix this, change the line reading ‘set theResult to value of text field "Full name:"’ so that it reads: ‘set theResult to value of static text 2’.

• The “Probe Window.applescript” UI element script fails with an error alert, and it does not fully describe how to see its results.

The script should display a dialog describing how to see its results, but instead it fails with this error message: ‘Error: System Events got an error: Can’t make class into type specifier.’

To fix this:

a. Change the line reading ‘get every UI element whose class is not button and class is not static text and class is not scroll bar’ so that it reads ‘get every UI element where it is not button and it is not static text and it is not scroll bar’.

b. Change the line reading ‘get properties of every UI element whose class is not button and class is not static text and class is not scroll bar’ so that it reads ‘get properties of every UI element where it is not button and it is not static text and it is not scroll bar’.

c. Change the line beginning ‘display dialog …’ should to the following, so that an inexperienced user will understand what to do in order to see the results: ‘display dialog "The 'Probe Window’ script was executed. It uses UI element scripting to get information about the top window in the Finder. To see the results, run it in Script Editor, choose Window > Log History, and click the Replies tab."

• The “Probe Menu Bar.applescript” UI element script works, but it does not fully describe how to see its results.

To fix this, the line beginning ‘display dialog …’ should be changed to the following, so that an inexperienced user will understand what to do in order to see the results: ‘display dialog "The 'Probe Menu Bar' script was executed. It uses UI element scripting to generate lists of every menu item of every menu for every application currently running. To see the results, open the script in Script Editor, choose Window > Log History and open the Replies pane, and run the script."

3. Why do some of my GUI Scripting scripts present authorization alerts?

For security reasons, macOS has long presented an alert requiring the user to grant access with a password before it will permit any script runner or applet to run a GUI Scripting script to monitor or control another application. In Mac OS X Mavericks 10.9 or later, the alert requires the user to grant access manually for each script runner or applet separately in the Accessibility list in the Security & Privacy preference's Privacy tab. When the user clicks the alert's System Preferences button, the Accessibility list is automatically opened and updated with a checkbox for the script runner or applet seeking access even if the Privacy pane is locked. The user must unlock the Privacy pane with a password in order to select the checkbox to grant access. The script runner does not present the alert again for a script runner or applet that was previously granted or denied access. Users can avoid the interruption of the initial alert by adding a script runner or applet to the Accessibility list and granting access with a password in advance. AppleScript may also present its own error alert separately, which should be dismissed before responding to the macOS alert

Starting with macOS Mojave 10.14, Apple has implemented additional authorization requirements. These requirements are not limited to GUI Scripting but also affect script runners and applets running AppleScript scripts generally, as well as applications that access data or other applications without using AppleScript at all. In some cases, more than one alert may be presented in succession. At this writing, Apple has not documented these requirements. The following summaries are based on our testing.

A new Automation list in the Privacy tab is updated when a user clicks OK in an alert presented by a script runner or applet that is attempting to control another application for the first time using AppleScript generally. The Automation list is updated automatically with the name of the script runner or applet together with a separate checkbox that is automatically selected to grant authorization to control the target application, even if the Privacy pane is locked, so the interruption is minimal. However, script runners and applets cannot be added to the list in advance, so the initial interruption cannot be avoided. After a script runner or applet is added to the list, the user can turn authorization to control the target application on or off manually by toggling its checkbox even if the Privacy pane is locked. A password is not required for any of this.

In addition, one of several named application lists, such as Contacts and Reminders, is updated when a user clicks OK in an alert presented by a script runner or applet that is attempting to access sensitive data managed by the named application for the first time. The named application list is updated automatically with a checkbox that is automatically selected to grant access to the requesting script runner or applet, even if the privacy pane is locked, so the interruption is minimal. However, script runners and applets cannot be added to the list in advance, so the initial interruption cannot be avoided. After a script runner or applet is added to the list, the user can turn access to the data on or off manually by toggling its checkbox even if the Privacy pane is locked, but another alert will advise that the change won't take effect until the script runner or applet is quit. A password is not required for any of this. Importantly, if the user denies access in the initial alert, the alert will never be presented again, and there appears to be no simple way at this time to grant access manually thereafter.

Finally, applications can be added manually to a new Full Disk Access list in the Privacy tab to access a variety of sensitive data, including data managed by Mail, Messages, Safari and other applications that are not included in the named application lists. An application can be granted access only manually, and only when the Privacy tab is unlocked with a password, in the Full Disk Access list. After an application is added to the list, the user can turn access on or off manually with a password by toggling its checkbox. If the application is running at the time, it must be quit before the change in access will take effect.

4. Why do GUI Scripting scripts start with nested double tell blocks?

GUI Scripting scripts usually start with a 'tell application' statement followed by a nested 'tell process' block. These statements can be generated automatically from UI Browser's AppleScript menu using the Tell Block Wrapper (Short) or Tell Block Wrapper (Safe) commands.

It is easy to assume that this is just some magic incantation and stop thinking about it, but in fact it is an example of standard AppleScript technique.

You will understand GUI Scripting better if you realize that the 'process' in the second, nested statement is simply an AppleScript object implemented by the System Events application. It is the same as any other AppleScript object, complete with elements, properties, and commands. For example, when you script the Finder, you might tell the Finder to tell the front window to get its position. In such a script, the Finder is the targeted application, the front window is an AppleScript object known to the Finder, and its position is a property of the window object. When you script System Events, you might tell System Events to tell process "Finder" to tell its front window to get its position. In such a script, System Events is the targeted application, the Finder process is an AppleScript object that System Events understands, the Finder's front window is an AppleScript object that the Finder understands, and its position is a property of the window object.

5. My script seems to get ahead of the target application's UI. Any suggestions?

Try the 'delay' command. Under Mac OS X 10.3 (Panther) or later, however, GUI Scripting is so much faster than it was with the Jaguar System Events public beta that you probably won't need to do this very often.

6. How do I script an Application's palettes?

Scripts that read or manipulate UI elements in a target application require special design to handle palettes, because a palette is automatically hidden when another application is activated. If you switch to Script Editor to run a script, for example, the target application's palettes become invisible and thus inaccessible to your script. Your script should therefore be written so that it activates the target application before attempting to read or manipulate palettes. In this way, the target application remains active and visible, and all of its UI elements can be scripted.

The same design principle applies to sheets, utility windows, inspectors, and some other kinds of windows that disappear when the application is not frontmost on the screen.

UI Browser is particularly helpful in these situations, because its hot keys allow you to select UI elements in a window while the target application is in front. In addition, almost every control in UI Browser can be used while you hold the Command key down to keep UI Browser in the background so that a palette will remain visible.

7. How do I script the menu bar using GUI Scripting?

Apple changed the way the menu hierarchy is exposed in Mac OS X 10.3 (Panther). Furthermore, the terminology in Panther differed from that in Jaguar, and, to confuse matters still further, in Panther the terminology differed as between the Accessibility API and GUI Scripting. Things became more logical sometime after Panther.

In Jaguar, the children of 'menu bar 1' were 'menu' elements, and each 'menu' element was conceptually a submenu having one or more 'menu item' elements as children. Each 'menu item' element represented a command on which you could click. Some 'menu item' elements had a single child element of the same name that was a 'menu' element, that is to say, an hierarchical submenu containing one or more 'menu item' elements. The 'menu item'/'menu' pairings continued down the hierarchy if there are submenus.

In Panther, a new level in the hierarchy was introduced immediately below the menu bar, called 'menu title' in the Accessibility API and 'menu bar item' in GUI Scripting. Apple did this to make the hierarchy that is exposed in the Accessibility API and in GUI Scripting correspond exactly to the underlying reality in the system. Thus, in Panther, the children of 'menu bar 1' became 'menu title' elements (addressed as 'menu bar item' elements in GUI Scripting). Conceptually, the 'menu bar' was actually a 'menu' containing one or more menu items, each of which was known as a 'menu title' or 'menu bar item' element. Each 'menu title' ('menu bar item') element had a single 'menu' element child of the same name, which was the submenu holding 'menu item' elements at the next level down the hierarchy. If there were submenus below that, they followed the same 'menu item'/'menu' pairings as in Jaguar.

Now, the Accessibility and GUI Scripting terms are the same, a 'menu bar item' is a 'menu bar item', and all is right with the world.

Well, not quite. For the past several releases of macOS, menus close automatically when another menu is opened and when another application comes to the front. This causes many difficulties for GUI Scripting, some of which are essentially insurmountable. For example, UI Browser cannot generate AppleScript statements referring to a menu item in an open menu, because opening UI Browser's AppleScript menu always causes the target application's menu to close. One way to work around this problem is to make a note of the UI element hierarchy displayed in the Path view while a target application's menu is open, then write a GUI Scripting reference to it manually.

8. How do I write scripts to send keyboard shortcuts?

Choose a target application in UI Browser, select its application element (or any other element) in UI Browser's browser view, open the Keystrokes drawer, type the desired key (taking care to use a lower-case character instead of an upper-case character, in the case of alphabetic characters), and check the Command key checkbox. Then open the AppleScript menu and scroll down to see the 'Send Defined Keystroke' command highlighted. This will open the Edit AppleScript window with the desired GUI Scripting statement selected and ready to copy or drag to a script editor. If you select appropriate UI Browser preferences, the generated script will already be on the clipboard, or it will be sent directly to your chosen script editor and inserted at the insertion point.

There is also a 'Send Defined Key Code' command in the AppleScript menu, which enables you to send function keys as well as character keys. The appropriate key code is displayed in the Keystrokes drawer when you press the desired function key (or any other key).

Keystroke and key code commands are always sent to the active, or focused, UI element in the target application. In the case of keyboard shortcuts, they are always sent to the target application itself, if it recognizes that shortcut. When sending keystrokes or key codes, it is never necessary to send it to a specific UI element.

The 'keystroke' command should be sent with a 'command down' parameter in a list created using AppleScript's curly braces syntax, like this: 'keystroke "n" using {command down}'. You can send the 'keystroke' command with multiple modifier keys separated by commas in the list, for keyboard shortcuts that take multiple modifier keys, like this: 'keystroke "n" using {shift down, command down}'. The character should always be lower case; if you use an upper case letter, GUI Scripting will think that the Shift key is down and the application will not respond as you intend.

In GUI Scripting, you can use the 'keystroke' command to send a string of characters, as long as they are all lowercase or all uppercase. UI Browser's Keystrokes drawer only lets you send one character at a time, because this is the way the Accessibility API works.

9. Why are the names of some UI elements different in a script from their names in UI Browser?

The names of some UI elements as they appear in UI Browser's main browser view differ from their AppleScript names as they appear in scripts generated using UI Browser's AppleScript menu. This is because the AppleScript terminology used in Apple's System Events dictionary for GUI Scripting is not identical to the terminology used by the Accessibility API. In UI Browser, we use the Accessibility names except when generating AppleScript statements, because UI Browser is a developer tool for the Accessibility API as well as a tool for GUI Scripting.

10. Why do some applications require a process name that is different from the application name?

In GUI Scripting, you usually first 'activate' the target application, and you then use nested 'tell' blocks to tell the System Events application to send commands to the target application's process. The 'activate' command is sent to the application itself, as in 'activate application "Adobe Photoshop CS2.app"'. In the inner 'tell' block of the nested 'tell' blocks, you tell System Events to tell the application's process to perform a command. In the case of a few applications, the name of the process is not the same as the name of the application; for example, 'tell process "Photoshop"'.

UI Browser knows which applications have different application and process names. For this reason, the safest course is to use UI Browser's AppleScript menu to generate 'tell' blocks for GUI Scripting. UI Browser always gets the names right.

11. Will my GUI Scripting scripts work in Mac OS X 10.2 (Jaguar) or earlier?

As a matter of historical interest, No, your GUI Scripting scripts will not work in Jaguar or earlier unless the computer includes some special beta supporting software that Apple first made available for testing in one of the earlier dot releases of Jaguar.

The scripts generated by UI Browser require Apple's GUI Scripting technology for AppleScript. Starting with Mac OS X 10.3 (Panther), the required supporting software was installed automatically, by default, on every Macintosh computer. Apple successfully concluded the GUI Scripting public beta test for Jaguar, and the beta version of the underlying GUI Scripting software for Jaguar is no longer available.

In the unlikely event that you have the Jaguar beta software and want to write scripts for it, you need to be aware that the AppleScript dictionary for GUI Scripting in the beta System Events application was somewhat different from the release version. For example, the beta dictionary defined 'check box' with a space for Jaguar, but in the final version for Panther it defines 'checkbox' without a space. Since the Jaguar version was beta software and is no longer officially available from Apple, UI Browser 1.4 and newer generates scripts for a 'checkbox' using the Panther terminology. If you are still using Jaguar with Apple's beta GUI Scripting software, edit generated 'checkbox' scripts to add a space by hand. Bear in mind that such a script may not work when it is run under Panther or newer.

Another example is the 'keystroke' command when it is used with modifier keys. The correct form for Jaguar was 'keystroke "n" with {command down}'; for Panther and newer, this was changed to 'keystroke "n" using {command down}'.