ISA provides extended screen recognition capabilities by automatically finding UI elements and shapes.

ISA helps you automate applications with limited or no automation API, such as Citrix. Also, you can use this option to automate applications that are otherwise slow, such as Remote Desktop applications, SAP, and others. In general, you can use this option for any application for which the application tree is incorrectly populated or is otherwise difficult to create a finder for.

Each recognized UI element contains several attributes that you can use in your finders.

Common attributes

• der_x, der_y, der_width,der_height: Coordinates and size of the element.

• lt_16_5, rb_15_4: Calculated numbers relative to the element's left top and right bottom coordinates. Element coordinates might change in different program state. For example, selected option with bold text can have different coordinates than the cleared option, and therefore finders might work incorrectly. This property eliminates coordinate change issues and can be used to create reliable finders. If a UI element has a name property, it is used in a finder by default when you insert an action step. Otherwise, property lt_16_5 is used in a finder.

The following table lists supported UI elements.

UI element name	Widget name	Element-specific attributes	Notes
Dialog box, frame, pane, and other	container	N/A	A general parent element that contains other elements. For example, an entire dialog box or a form inside this dialog box can be a container. One of the special containers that Kofax RPA can recognize is a table.
Element	element		A general UI element that does not contain other elements.
Icon	icon	label	A graphical element that does not contain any text.
Table	table		Tables are represented in the application tree as separate elements with the following restrictions: merged cells are not recognized and headers are the same as regular cells. Each cell is defined as an item in the application tree. Items can contain other UI elements, such as check boxes, text boxes, and other.
Row	row		A row in the table tree.
Item	item		An item within a row in the table tree.
Text box	textbox	label	A text field.
Check box	checkbox	label	A check box.
Option button	radiobutton	label	An option button or radio button.
Text and text label	linklabel	name	Either a standalone text element, such as dialog box subtitle, or a UI element label, such as a check box label, text box label, and so on. If a text is recognized as a UI element label, the UI element will contain a label property with the label name as its value. In the following example, textbox element will contain label="Name" property.

ISA tips and tricks

• Recognition results depend on many factors, such as screen fonts, background and foreground color, text size, and so on. If you want to improve recognition results, try using different color schemes in your interface, such as use regular monospaced black-colored fonts on a bright background and larger font size.
• Selected text with inverted colors in a text field might not be correctly recognized. To improve recognition results, remove the selection from the text field (such as by clicking somewhere else) before extracting its value.
• When you work with dynamic content forms and text fields that might be recognized differently in different state, use the Freeze Tree step to hold the application tree state and improve recognition results.
• If you want to extract value from an element, use the Extract Value From action as you would do when using Windows automation API. If the element's value is not recognized in ISA mode, try Extract Text From Image Into action.
• You can edit extended OCR settings in the ocr.cfg file. See Extended OCR Settings for more information.

• You can train Tesseract to recognize your character set using either TTF fonts or UI screen shots. See Train Tesseract for more information.

Important If you connect remotely to a Windows automated device via Remote Desktop Protocol (RDP), resolution and color depth depend on the network connection speed between your computer and remote device. Different connection speed in different robot runs can lead to issues if robots use Image Finder and Intelligent Screen Automation (ISA). For Image Finder and ISA it is very important to always receive the same picture (pixel-by-pixel) in the same application state. Always use the same explicitly-specified resolution and color depth parameters (g: desktop geometry (WxH) and a: connection color depth) for the RDP connection in the Open step. For example:

rdp://user1:MyPassword@MyDesktop?g=640x480&a=16

Change or add UI recognition language

By default, ISA can automate the English language UI. You can add more languages for UI recognition, using the procedure similar to changing the default OCR language. Note that using more than one language simultaneously for screen recognition slows down robot execution and deteriorates recognition results.

1. Download the .traineddata file for the required language from the https://github.com/tesseract-ocr/tessdata. For example, the file for the Japanese language is jpn.traineddata.
2. Copy downloaded trained data file to the appropriate folder:

   • On the Windows-based automated computer with installed Desktop Automation Service:

     DesktopAutomationService\lib\tessdata in the Desktop Automation Service installation directory. Example:

     C:\Program Files (x86)\Kofax RPA DesktopAutomation 10.6.0 x32\DesktopAutomationService\lib\tessdata

   • On the local Windows-based computer to use with built-in browser:

     nativelib\hub\windows-x32\<build number>\lib\tessdata* in the Kofax RPA installation directory. Example:

     C:\Program Files\Kofax RPA 10.6.0.0 x64\nativelib\hub\windows-x32\622\lib\tessdata

   • On the local Linux-based computer to use with built-in browser:

     nativelib/hub/linux-x64/<build number>/lib/tessdata in the Kofax RPA installation directory. Example:

     Kofax_RPA_10.6.0.0_x64/nativelib/hub/linux-x64/533/lib/tessdata

   * The build number is different in different versions of the program.

3. In the text editor, open the isa_v1.cfg file located in the lib directory.
4. In the ocr_language parameter either replace eng with jpn or, if you want to use more than one language, add jpn using the plus sign, such as ocr_language=eng+jpn. Save and close the file.
5. Restart the Desktop Automation Service for the changes to take effect.

To generate a widget tree for Windows applications, Kofax RPA uses the UI Automation framework built into Windows. Also, Kofax RPA provides some extended support for particular applications. If you experience any issues working with applications using Kofax RPA extensions, turn them off by clearing the selection in the Windows tree mode.

Excel

Turns on extended support for Microsoft Excel, such as to retrieve cells from the spreadsheet. Excel support in Kofax RPA is also extended with Built-in Excel Driver features.

Note When using Enter Text in the Excel on a remote device, do not leave the Excel spreadsheet in edit mode (the word Edit appears in the lower-left corner of the Excel program window) if you want to work with it in the following steps. To exit the edit mode, perform one of the following.

• Press ENTER using the Press Key step. Excel exits Edit mode and selects the cell directly below the current cell.
• Press TAB using the Press Key step. This stops Edit mode and selects the cell to the right of the current cell.
• Click a different cell.
• Press F2 using the Press Key step.

See Microsoft documentation for more information.

Internet Explorer

Turns on extended Internet Explorer support to retrieve DOM (Document Object Model) tree as it provides a more accurate result in the application tree. Note that Kofax RPA also provides a built-in chromium-based web browser to access websites. Note the following when considering using extended Internet Explorer support.

• If extended Internet Explorer support is enabled and the robot detects an Internet Explorer window:
  • A piece of Java-script is injected that traverses the Document Object Model (DOM) and creates a JSON representation. This runs in the context of the Internet Explorer page.
  • The JSON is placed into an input tag with id "kapowDataElement" in the page itself.
  • The robot parses the JSON content and creates the sub-tree from that content.
• If extended Internet Explorer support is disabled, the UI Automation is used to get the object tree from Internet Explorer (this is the same tree that you get using Microsoft's Inspect.exe).

Java

In some situations the Java Access Bridge does not work and it can help to switch to legacy mode by clearing the selection of this option. The Java option can also have an influence on Java Applets in Internet Explorer.

SAP

SAP is handled differently from other Windows applications. Working with SAP depends on scripting API that might by slow. You can turn off SAP support by clearing this option.

Hidden

Specifies whether to extract the entire tree of an application. This option is selected by default. If you clear the selection, Kofax RPA skips elements that are reported as off-screen, such as list boxes or tables with many elements. Clearing the selection reduces the time needed to extract the tree.

Legacy Depth+Index

Adds the "depth" and "index" attributes to the application tree that were used in Kofax RPA 10.2. If you have robots created in Kofax RPA version 10.2 that use component finders with "depth" and "index" attributes, select Legacy Depth+Index for the robots to work in Kofax RPA 10.3 and later.

• Max Depth: Sets the maximum number of nested elements each node shows for all application windows in the view.
• Max Siblings: Sets the maximum number of sibling elements each node shows for all application windows in the view.

  Specify 0 for no restrictions.

Excel API Retry Count

Specifies the number of retries when accessing the Excel window via Windows API. You can increase the number if Excel response is slow.

Excel API Retry Wait Time

Specifies the time in milliseconds between retries when accessing the Excel window via Windows API.