Crawling a Popup Menu
You can discover all pages linked directly to a popup menu without continually crawling from these pages.
We do not wish to continue crawling from these pages.
- Add a step with the Load Page action that loads the main page.
- Add a new step and choose the Crawl Pages action.
- Select the menu bar as named tag.
- Notice that the "Automatically Handle Popup Menus" option on the Crawling tab is checked.
- On the Rules tab, add a Crawling Rule saying that for "All URLs" we "Do Not Crawl", but "Output the Page".
- After the step with the Crawl Pages action, add steps to handle each page, e.g. by extracting information into returned variables.