With Schedule, you can set up a cron job to run at specified time intervals to automatically extract data. You can set up more than one schedule depending on how often you want to source data. Schedules can be set as recurring if they need to repeat at fixed intervals.
Fig: Scheduling a crawler
To set up a schedule
- Navigate to the Schedule menu in the Report > Configuration section
- Add a Start Date and Time for the schedule. This is the first instance the schedule will run
- Next in the Repeat menu, choose
- Never if the schedule need not repeat
- Hourly, Daily, Weekly or Monthly depending on how often you need to pull data
- The next two fields, Repeats Every and Repeats On is dependent on how frequently the schedule should repeat. Use following guide while selecting these fields
- Next, choose if the schedule should repeat forever or terminate at a fixed date.
- Give your schedule an apt name in case you have multiple schedules
- Lastly, set Parameters. Parameters are optional in most cases but can be used to change the behavior of the crawler. You can ignore this field completely if the parameter option reads “No Parameter”. Reach out to your account manager how to or if you should use parameters.
💡More on Parameters
Parameters are configured as per customer requirement while the report is initially set up. The parameter values can be changed on the fly during execution. For example, assume a crawler collects product data for a specific category from Amazon. If the crawler is set up to take in a category as an input then the customer can control which category of product is collected for every run.
Topics in this section: