Crawling a Popup Menu

You can discover all pages linked directly to a popup menu without continually crawling from these pages.

We do not wish to continue crawling from these pages.

  1. Add a step with the Load Page action that loads the main page.
  2. Add a new step and choose the Crawl Pages action.
  3. Select the menu bar as named tag.
  4. Notice that the "Automatically Handle Popup Menus" option on the Crawling tab is checked.
  5. On the Rules tab, add a Crawling Rule saying that for "All URLs" we "Do Not Crawl", but "Output the Page".
  6. After the step with the Crawl Pages action, add steps to handle each page, e.g. by extracting information into returned variables.