This document provides an overview of the classes and methods available in the uiautomation.py module for UI automation tasks.

Table of Contents

1. ScreenInteraction
2. BitmapInteraction
3. WindowInteraction
4. Utility Functions

ScreenInteraction

The ScreenInteraction class provides methods for interacting with the screen, including taking screenshots, moving the mouse, simulating keyboard input, and performing OCR (Optical Character Recognition).

Methods

save_screenshot(filename=None)

Saves a screenshot of the entire screen.

• filename (optional): The name of the file to save the screenshot. If not provided, a timestamp-based name will be used.
• Returns: The filename of the saved screenshot.

move_mouse(x, y, click=None, delay=1, takescreenshot=False)

Moves the mouse to specified coordinates and optionally performs a click.

• x, y: The target coordinates.
• click: “left” for left click, “right” for right click, or None for no click.
• delay: Time to wait after moving the mouse (in seconds).
• takescreenshot: If True, takes a screenshot after moving the mouse.
• Returns: Filename of the screenshot if takescreenshot is True, otherwise None.

simulate_typing(text, delay=0.01)

Simulates typing the given text.

• text: The string to be typed.
• delay: Time delay between each character (in seconds).

press_key(key='f5')

Simulates pressing a keyboard key.

• key: The key to be pressed (default is ‘f5’).

scroll_mouse(direction, amount, duration=1)

Simulates mouse scrolling.

• direction: ‘up’ or ‘down’.
• amount: Number of scroll units.
• duration: Time over which to perform the scrolling (in seconds).

capture_screen()

Captures the entire screen.

• Returns: A PIL Image object of the screen.

perform_ocr(image)

Performs OCR on the given image.

• image: A PIL Image object.
• Returns: A dictionary containing OCR results.

find_phrase_coordinates(ocr_result, target_phrase, similarity_threshold=80)

Finds the coordinates of a phrase in OCR results.

• ocr_result: The result from perform_ocr().
• target_phrase: The phrase to find.
• similarity_threshold: Minimum similarity score to consider a match.
• Returns: A dictionary with phrase information and coordinates, or None if not found.

save_ui_element(result, filename, border=8)

Saves a screenshot of a UI element.

• result: The result from find_phrase_coordinates().
• filename: The name of the file to save the screenshot.
• border: Additional border around the element (in pixels).

find_text_on_screen(target_phrase)

Finds and saves a screenshot of text on the screen.

• target_phrase: The text to find.
• Returns: The filename of the saved screenshot, or an empty string if not found.

save_ui_item(x, y, w, h, filename, border=8)

Saves a screenshot of a specific area on the screen.

• x, y: The top-left coordinates of the area.
• w, h: The width and height of the area.
• filename: The name of the file to save the screenshot.
• border: Additional border around the area (in pixels).

get_font_size(size, req_size)

Determines the font size category based on pixel height.

• size: The pixel height of the font.
• req_size: The requested size category (“any”, “large”, “medium”, or “small”).
• Returns: The determined size category.

find_str_on_screen(target_str, size="any", border=8)

Finds and saves screenshots of a string on the screen.

• target_str: The string to find.
• size: The font size category to look for (“any”, “large”, “medium”, or “small”).
• border: Additional border around found items (in pixels).
• Returns: Filename of the first found item, or a tuple of the first filename and all filenames if multiple are found.

is_there(filename, timeout=5, confidence=0.8)

Checks if an image is present on the screen.

• filename: The filename of the image to look for.
• timeout: Maximum time to search (in seconds).
• confidence: Minimum confidence level for a match (0-1).
• Returns: True if the image is found, False otherwise.

BitmapInteraction

The BitmapInteraction class provides methods for interacting with bitmap images on the screen.

Methods

wait_for(filename, timeout=20, confidence=0.8)

Waits for an image to appear on the screen.

• filename: The filename of the image to wait for.
• timeout: Maximum time to wait (in seconds).
• confidence: Minimum confidence level for a match (0-1).
• Returns: True if the image appears within the timeout, False otherwise.

click(filename)

Clicks on an image on the screen.

• filename: The filename of the image to click.
• Returns: True if the image was found and clicked, False otherwise.

WindowInteraction

The WindowInteraction class provides methods for interacting with windows and browser tabs.

Methods

bring_window_to_front(window_title)

Brings a window with the specified title to the front.

• window_title: The title of the window to bring to front.

resize_active_window(width, height)

Resizes the currently active window.

• width, height: The desired dimensions of the window.

maximize_active_window()

Maximizes the currently active window.

minimize_active_window()

Minimizes the currently active window.

get_monitor_info(window)

Gets information about the monitor a window is on.

• window: The window object.
• Returns: A dictionary with monitor information.

move_active_window_top_right(stay_on_current_monitor=True)

Moves the active window to the top right corner of the screen or monitor.

• stay_on_current_monitor: If True, stays on the current monitor.

move_active_window_bottom_left(stay_on_current_monitor=True)

Moves the active window to the bottom left corner of the screen or monitor.

• stay_on_current_monitor: If True, stays on the current monitor.

save_clipboard_text_to_file(file_path)

Saves the current clipboard content to a file.

• file_path: The path where the file should be saved.

select_browser_tab(tab, window, strict=False)

Selects a browser tab.

• tab: The name of the tab to select.
• window: The browser window object.
• strict: If True, requires an exact match for the tab name.
• Returns: True if the tab was found and selected, False otherwise.

get_browser_window(target_tab, to_print=True)

Gets a browser window object for a specific tab.

• target_tab: The name of the tab to find.
• to_print: If True, prints debug information.
• Returns: The window object if found, None otherwise.

browser_item_found(main_window, auto_id, control_type)

Checks if a specific item exists in a browser window.

• main_window: The browser window object.
• auto_id: The auto ID of the item to find.
• control_type: The type of control to find.
• Returns: True if the item is found, False otherwise.

browser_item_coordinates(main_window, auto_id, control_type)

Gets the coordinates of a specific item in a browser window.

• main_window: The browser window object.
• auto_id: The auto ID of the item to find.
• control_type: The type of control to find.

browser_update_edit_field(main_window, auto_id, text)

Updates the text in an edit field in a browser window.

• main_window: The browser window object.
• auto_id: The auto ID of the edit field.
• text: The text to set in the edit field.

browser_press_button(main_window, auto_id, title)

Presses a button in a browser window.

• main_window: The browser window object.
• auto_id: The auto ID of the button.
• title: The title of the button.

get_current_browser_tab_title()

Gets the title of the current browser tab.

• Returns: The title of the current browser tab.

click_ui_button(button, window)

Clicks a button in a UI window.

• button: The text of the button to click.
• window: The window object containing the button.

Utility Functions

generate_random_number_string(length=4)

Generates a random string of numbers.

• length: The length of the string to generate.
• Returns: A string of random numbers.

Predefined Instances

The module provides predefined instances of each class for convenience:

• screen: An instance of ScreenInteraction
• bitmap: An instance of BitmapInteraction
• window: An instance of WindowInteraction

These can be used directly after importing the module, e.g., screen.save_screenshot().