ISA provides extended screen recognition capabilities by automatically finding UI elements and shapes.

ISA helps you automate applications with limited or no automation API, such as Citrix. Also, you can use this option to automate applications that are otherwise slow, such as Remote Desktop applications, SAP, and others. In general, you can use this option for any application for which the widget tree is incorrectly populated or is otherwise difficult to create a finder for.

Each recognized UI element contains several attributes that you can use in your finders.

Common attributes

• der_x, der_y, der_width,der_height: Coordinates and size of the element.

• lt_16_5, rb_15_4: Calculated numbers relative to the element's left top and right bottom coordinates. Element coordinates might change in different program state. For example, selected option with bold text can have different coordinates than the cleared option, and therefore finders might work incorrectly. This property eliminates coordinate change issues and can be used to create reliable finders. If a UI element has a name property, it is used in a finder by default when you insert an action step. Otherwise, property lt_16_5 is used in a finder.

The following table lists supported UI elements.

UI element name	Widget name	Element-specific attributes	Notes
Dialog box, frame, pane, and other	container	N/A	A general parent element that contains other elements. For example, an entire dialog box or a form inside this dialog box can be a container. One of the special containers that Kapow can recognize is a table.
Button	button	name	Regular buttons with defined borders.
Icon	icon	label	A graphical element that does not contain any text.
Table	table		Tables are represented in the widget tree as separate elements with the following restrictions: merged cells are not recognized and headers are the same as regular cells. Each cell is defined as an item in the widget tree. Items can contain other UI elements, such as check boxes, text boxes, and other.
Text box	textbox	label	A text field.
Check box	checkbox	label
Option button (or radio button)	radiobutton	label
Text and text label	linklabel	name	Either a standalone text element, such as dialog box subtitle, or a UI element label, such as a check box label, text box label, and so on. If a text is recognized as a UI element label, the UI element will contain a label property with the label name as its value. In the following example, textbox element will contain label="Name" property.

ISA tips and tricks

• Recognition results depend on many factors, such as screen fonts, background and foreground color, text size, and so on. If you want to improve recognition results, try using different color schemes in your interface, such as use regular monospaced black-colored fonts on a bright background and larger font size.
• Selected text with inverted colors in a text field might not be correctly recognized. To improve recognition results, remove the selection from the text field (such as by clicking somewhere else) before extracting its value.
• When you work with dynamic content forms and text fields that might be recognized differently in different state, use the Freeze Tree step to hold the widget tree state and improve recognition results.
• If you want to extract value from an element, use the Extract Value From action as you would do when using Windows automation API. If the element's value is not recognized in ISA mode, try Extract Text From Image Into action.
• You can edit extended OCR settings in the ocr.cfg file. See Extended OCR Settings for more information.

Change or add UI recognition language

By default, ISA can automate the English language UI. You can add more languages for UI recognition, using the procedure similar to changing the default OCR language. Note that using more than one language simultaneously for screen recognition slows down robot execution and deteriorates recognition results.

1. Download the .traineddata file for the required language from the https://github.com/tesseract-ocr/tessdata. For example, the file for the Japanese language is jpn.traineddata.
2. Copy downloaded trained data file to the appropriate folder:

   • On the Windows-based automated computer with installed Device Automation Service:

     DeviceAutomationService\lib\tessdata in the Device Automation service installation directory. Example:

     C:\Program Files (x86)\Kapow DeviceAutomation 10.3.0 x32\DeviceAutomationService\lib\tessdata

   • On the local Windows-based computer to use with built-in browser:

     nativelib\hub\windows-x32\<build number>\lib\tessdata* in the Kofax Kapow installation directory. Example:

     C:\Program Files\Kapow 10.3.0.0 x64\nativelib\hub\windows-x32\622\lib\tessdata

   • On the local Linux-based computer to use with built-in browser:

     nativelib/hub/linux-x64/<build number>/lib/tessdata in the Kofax Kapow installation directory. Example:

     Kapow_10.3.0.0_x64/nativelib/hub/linux-x64/533/lib/tessdata

   * The build number is different in different versions of the program.

3. In the text editor, open the isa.cfg file located in the lib directory.
4. In the ocr_language parameter either replace eng with jpn or, if you want to use more than one language, add jpn using the plus sign, such as ocr_language=eng+jpn. Save and close the file.
5. Restart the Device Automation service for the changes to take effect.

To generate a widget tree for Windows applications, Kofax Kapow uses the UI Automation framework built into Windows. Also, Kapow provides some extended support for particular applications. If you experience any issues working with applications using Kapow extensions, turn them off by clearing the selection in the Windows tree mode.

Excel

Turns on extended support for Microsoft Excel, such as to retrieve cells from the spreadsheet. Excel support in Kofax Kapow is also extended with built-in Excel functionality.

Internet Explorer

Turns on extended Internet Explorer support to retrieve DOM (Document Object Model) tree as it provides a more accurate result in the widget tree. Note that Kofax Kapow also provides a built-in chromium-based web browser to access websites.

Java

In some situations the Java Access Bridge does not work and it can help to switch to legacy mode by clearing the selection of this option. The Java option can also have an influence on Java Applets in Internet Explorer.

SAP

SAP is handled differently from other Windows applications. Working with SAP depends on scripting API that might by slow. You can turn off SAP support by clearing this option.

Hidden

Specifies whether to extract the entire widget tree of an application. This option is selected by default. If you clear the selection, Kapow skips elements that are reported as off-screen, such as list boxes or tables with many elements. Clearing the selection reduces the time needed to extract the tree.

Legacy Depth+Index

Adds the "depth" and "index" attributes to the widget tree that were used in Kofax Kapow 10.2. If you have robots created in Kapow version 10.2 that use component finders with "depth" and "index" attributes, select Legacy Depth+Index for the robots to work in Kofax Kapow 10.3 and later.

• Max Depth: Sets the maximum number of nested elements each node shows for all application windows in the view.
• Max Siblings: Sets the maximum number of sibling elements each node shows for all application windows in the view.

  Specify -1 for no restrictions.