HTTP URL Monitoring

In an environment, where the downtime of any website applications and services can cause a negative impact on the business performance, problems must be accurately identified and solved.

Applications Manager acts as a continuous URL monitoring service that keeps constant watch over a specified URL or website page. They test the website applications and web services to ensure that they are functioning properly. URL monitors verify the availability of specified, addressable, standard HTTP and HTTPS URLs. They scan the HTTP and HTTPS pages looking for a predefined keyword to check whether the website is available.

• HTTP(s)-URLs
• HTTP(s)-URL Sequence

The difference between the two types of monitoring is that URL Monitoring monitors single HTTP and HTTPS URL, whereas URL sequence monitors a set of HTTP and HTTPS URLs in sequence. Also, any HTTP and HTTPS URL can be monitored using URL Monitoring. Please go through the following sections to know about the configuration details.

Note: For monitoring, URL should be accessible from the machine where Applications Manager is running.

HTTP-URLs

Prerequisites for monitoring URL metrics: Click here

Using the REST API to add a new URL monitor: Click here

To configure for URL monitoring, follow the given steps:

1. Select New Monitor. Choose HTTP-URLs.
2. Provide any display name for the HTTP-URL monitoring.
3. Provide the HTTP/HTTPS URL address you want to monitor.
4. Provide the Polling Interval (in minutes) for which Applications Manager updates the status of the monitor.
5. Enter the Timeout (in seconds) for which the URL monitor should wait for a page to complete downloading before timing out.
6. Use the Add Advanced Web Monitors field to add other advanced web monitors like SSL/TLS Certificate Monitor, Webpage Analyzer, DNS Monitor, Brand Reputation, and Website Content Monitor.
7. If you are adding a new monitor from an Admin Server, select a Managed Server.

HTTP Configurations: Apart from the basic URL Monitoring, Applications Manager also provides you with advanced options that furnish effective and more flexible URL Monitoring. This is optional and you need to choose these options only if the HTTP/ HTTPS URL requires Form-based authentication.

7. Choose a Form Submission Method between the POST and GET, which are the two types of methods for any HTTP/HTTPS URL. If you choose POST, select Payload Type for 'POST' form submission.
8. Provide the request parameters, if any. The request parameters must be provided as name=value pairs for POST and GET methods. For example, if you want to monitor a URL like, https://apm-prod-server:9090/showresource.do?haid=1&type=UrlMonitor&method=getMonitorForm, then provide "?haid=1&type=UrlMonitor&method=getMonitorForm" as request parameter.
9. Enter the credential details like user name and password for authentication, or select the required credentials from the Credential Manager list after enabling the Select from Credential list option. You can also authenticate through SAML ADFS Authentication by selecting the required credentials added using SAML Authenticator.
10. Specify the response code and choose the appropriate condition for which the error should be notified by specifying the criteria under Error if Response Code field. By default, it is greater than 200.
11. Enable the "If monitor detects error, re-try immediately to verify error" checkbox, if the monitoring is to be automatically performed whenever it detects an error, i.e. when an error is detected, the monitor will immediately be scheduled to run again once.
12. Sometimes you may want to customize the default HTTP request header. You have the option to customize the "User-Agent" field. Check the Custom HTTP(S) Headers and User Agent checkbox and enter the HTTP Header and User Agent in the text boxes. You may add multiple HTTP Headers and User Agents.
13. Choose the SSL version from the drop-down menu, if enabled. By default, it is set as Auto.
14. Enable the Client Certificate Authentication checkbox if and only if you have enabled Client Certificate Authentication for the server you want to monitor. Please note that only files of type "PKCS#12"(.p12) are supported. Also, provide the Client Certificate Passwordfor the certificate, if required.

    Note: When you upload two Client Certificates of same name, note that the previous one will be overwritten by the later.

15. Enter a keyword which when matches with the content in the URL must be notified of error in Error If Match field. Note: The search is case sensitive. Multiple keywords should be put within quotes. Otherwise, it will be treated as 'Any' one of the multiple keywords.

Content check: Additionally, you can choose to perform content check in the URL page. The URL monitoring searches the keyword in the content of the URL page to check the availability of the URL.

16. Specify the contents that should contain and/or should not contain while performing the check in the API response.
17. Enable the Do case sensitive keyword search checkbox if you wish to perform a content check for case-sensitive character strings.
18. Enable the Regular Expression checkbox if you wish to perform a content check based on regular expressions. (Refer Regex Guide for more info)

16. Click Add URL Monitor to initiate monitoring of the specified URL.

HTTP(s)-URL Sequence

The purpose of URL Sequence is to monitor multiple web pages of an online application. It checks pages with dynamically generated information enhancing interactive transactions such as logging into a login page, creating an account using a web form, instructing the application to perform some action, etc.

URL Sequencing starts with a specific URL such as the Login page and then followed by additional links/URL in it. The URL Sequence monitoring thus performs end-to-end verification of particular transactions helping you to troubleshoot any problems while monitoring.

Note:
If you are connecting to an URL using a proxy server, then you must configure proxy to initiate the URL sequence monitoring. Refer to the Configuring Proxy section of Performing Admin Activities, for further details.

To create an HTTP(s) URL Sequence monitor, follow the steps given below:

1. Go to New Monitor and click Add New Monitor.
2. Select HTTP(s) URL Sequence Monitor under Web Server/Services category.
3. Here you can create a URL Sequence monitor using any one of the following methods:
   • Import HAR File
   • Use Web Transaction Recorder

To modify an existing URL Sequence monitor, refer here.

Import HAR File

On selecting this option, you will be taken to the page where you have to import HAR file. To create a URL sequence monitor by importing HAR file, the user must generate a HAR file comprising of the required URL Sequence.

1. Import HAR file and view URL(s) recorded

1. Generate a HAR file. Click here to learn about generating HAR file
2. Enter a display name for your URL Sequence monitor.
3. Upload the generated HAR file. Maximum file size allowed is 60 MB.
4. Enter the required Poll Interval value in minutes. Default value is 10 minutes.
5. Click Upload File. This will take you to a page where you can see all the URLs recorded and modify them as required.

2. View and modify recorded URL(s)

• Display Name field contains the monitor name that was provided while uploading the HAR file.
• File Name contains the name of the file which was uploaded.
• Connection Timeout is the value of timeout for connection for each URL in the sequence.
• URL Listsection lists all the URLs that have been recorded. To delete URL(s),
  • Select the check boxes corresponding to them in the URL list.
  • Click on the Delete button on the right side below the list.
• URL Details section contains the information such as URL, display name, headers, HTTP Request method, etc., for the URL currently selected or clicked upon. It has two tabs - Basic and Advanced, present as a radio button below the URL List.

  Note: All changes done here will affect only the selected URL, not all URLs listed.

  • Basic:
    • URL field has the URL which is current selected. You can modify this if required.
    • Display Name is the name that will displayed for the URL in the list of the URLs in the URL Sequence monitor.
    • Headers field has the headers that are used by this URL for communication. Applications Manager will also be sending the same URLs while making a request to this URL.
    • HTTP request methodshows the URL request method, whether it is GET or POST. (PUT, PATCH, HEAD, DELETE URLs are not monitored)
      • If the URL request method is of type GET, then the request params used for the URL will be shown in the Request Parameters section. The content can be modified to include any extra data if required.
      • If the URL is of type POST, then the post data will be shown under the Payload section in FORM, TEXT, XML, or JSON format. All the sections can be modified to include any extra data if required.
  • Advanced:
    • Alert Configuration- You can add content check for a URL if required.
      • Generate Alert if following keywords are present: If the keywords added here are present in the URL’s web page then the monitor’s status will be made down and an alert will be generated. So the keywords here are usually the keywords that one would like the website not to contain.
      • Generate Alert if following keywords are not present: If the keywords added here are not present in the URL’s webpage then the monitor’s status will be made down and an alert will be generated. So the keywords here are usually the keywords that one would like the website to contain.
    • Basic authentication - The basic authentication details, if present in a URL will be automatically captured by Applications Manager. If you still wish to add authentication details, you can enter the credential details like user name and password for authentication, or select the required credentials from the Credential Manager list after enabling the Select from Credential list option for URL monitor. (To add new credentials under Credential Manager, refer here)
    • Response Code configuration - Configures the response code and the condition that you would like to be alerted for. By default you will be alerted for all URLs that return a response code above 200. You can modify them if required.
• Filter URLssection filters the URLs based on the following categories:
  • URL Extension (File Type) -- Filters the captured URLs based on the extension of the file type. By default, URL(s) will be filtered based on default file extension filters like .css, .js, .tff, etc.
  • Domain Components -- Filters the captured URLs based on the domains from which they originate (like abc.com or example.com).
  • Content Components -- Filters the captured URLs based on the “Content-Type” header in the response headers like image, video, audio, text and application.

  You can add new filter(s) to each of the above categories (URL Extension, Domain Component or Content Component) using the Add button at the bottom of each category list.

• Associate monitor to a monitor group: Choose the Monitor Group from the combo box with which you want to associate the URL Sequence monitor (optional). You can choose multiple groups to associate your monitor.

After performing all the above steps, click on “Add Monitor” to create the URL Sequence monitor.

Note: Only 25 URLs are allowed to be recorded. If the URLs exceed more than 25, then the monitor will be created with the first 25 URLs. Alternatively, you can also delete the URLs that are unnecessary using the Delete functionality explained earlier.

Use Web Transaction Recorder

Applications Manager's Web Transaction Recorder tool records all user online transactions in their exact sequence. These transactions will then be replayed at regular intervals of time and notifications will be sent when any error is detected.

Browse through the following topics to use the Transaction Recorder tool:

• Getting Started
  • Hardware/Software Requirements
  • Download and Installation
• Recording a New Transaction
  • Recording a Sequence
  • Editing captured URL's
  • Saving a transaction
• Modify an existing Transaction
• Monitoring URLs with Dynamic Parameters

Note:
We have set a limit of 25 URLs per URL Sequence monitor.

Getting Started

You must download and install the recorder tool to record the sequence of user transactions. Every step of the transaction will be captured and recorded. You also can modify or edit your transactions.

Hardware/Software Requirements

The Transaction Recorder is developed using the Microsoft .NET Platform version 2.0. This is the minimum software requirement. A minimum of 2 MB of disk space is also required for installing the software.

Download and Installation

Download the Web Transaction Recorder and start the installation process. The installation wizard takes you through a four step process after which the Recorder will be started:

1. A welcome screen is displayed, after which you will be asked for confirmation of the license requirements. Click Yes.
2. After confirmation, a screen to Chose Destination Location is displayed. The user is allowed to select the drive and directory where he wishes to install the Transaction Recorder. Click Browse and select the installation destination. Click Next.
3. Review the settings in the confirmation dialog and click Back if you wish to make changes. If not, click Next. The Recorder will be installed in your machine.
4. After the installation is completed, click Finish. Make sure the "Start Transaction Recorder" option is checked, if you would like to start the tool automatically after the installation is complete.

Recording a New Transaction

After installing, the Transaction Recorder can be accessed under All Programs in Windows Start Menu. You can record a new transaction following these steps:

1. Record the Sequence.
   • Click on the New button to start a new recording.
   • Enter the URL of the application you wish to monitor then press enter to load the corresponding web page.
   • Follow the sequence of steps that you wish to monitor.
2. Edit the captured URL's
   • After you have completed the sequence, click Preview & Save. A Login Credentials dialog box is displayed.
   • Provide the login credentials - Host Name, Port Number, User Name and Password. Check the Show Password checkbox to see the password
   • Click on Next to open the Preview Transactions screen. Here the user can view the recorded URLs and configure monitoring settings for each URL individually.
   • In the Preview Transactions screen, click the Filter URLlink on the top right corner to open a URL Filter Screen. URLs can be filtered based on the following three categories:
     • File Type / URL Extensions -- The captured URLs can be filtered based on their extensions. For example, .gif, png, jpg etc. for images, .js for scripts and more.
     • Content Type / MIME Type -- URLs can be filtered based on the MIME type like image, video, audio, text and application.
     • Domain Components -- You can also filter URLs based on the domains from which they originate. You have the option to include / exclude websites from a particular domain in the Domain Filter list.

       Note:
       These domains are not stored for future references. You have to re-enter such preferences if you re-start the tool.

   • You can add a new Extension, Content Type or Domain Component to the URL filter using the ADD button at the bottom of each list.
   • Delete URL: Click on a URL from the list of transactions and click the Delete URL link on the top right corner to delete it.
   • Basic Configuration: You can edit basic URL details like URL link, Display name, header and post data by selecting settings Basic option at the bottom of the list of recorded URLs.
   • Advanced Configuration:
     • Alert Configuration: You can configure the recorder such that alerts are generated when specific keywords are present/not present. Enter the keywords in the text boxes provided.
     • Basic Authentication Details: Enter Authentication details like username and password.
     • Response Code Configuration: You can configure the recorder to generate alerts if response time is greater than, equal to, not equal to, less than, less than or equal to a specified value.
   • After making all the required changes, click next to go to the save transactions screen.
3. Saving the transaction: In the save transactions screen, you can configure or edit the following fields:
   • URL sequence name: The wizard generates a sequence name. You can edit the default sequence name.
   • Poll interval: Poll interval is the time in minutes between two consecutive attempts to check for each recorded URL.
   • Host name
   • Port number
   • Username and Password. (The username and password used for Applications Manager with administrative privilege should be used).
4. Click Save to save the changes.

Modify an existing transaction

You can modify an existing transaction

1. Click Re-record.
2. Record the Sequence.
3. Edit the captured URLs
4. Select the transaction which needs to be modified from the drop-down menu and the click Fetch URLs.
5. Click next to perform a side-by-side comparison of the list of old URL's and the list of newly recorded URL's.
6. Click Update to update the existing transaction with the currently recorded sequence in the server.

Monitoring Dynamic URLs:

Suppose your login page has hidden parameters that needs to be submitted while logging into your application and the value for the hidden parameter is different for each login request. In this case, the hidden parameter value in the login page needs to be fetched dynamically and passed to the subsequent URL.

For monitoring these kind of dynamic URL Sequences, you can use $<Parameter name> [eg., $sessionID = 12] in the Request Parameter text box of the URL. So, while monitoring the URL Sequence, $< Parameter name> will fetch the dynamic values and pass it to the next URL in the sequence.

While using the Recorder tool, you cannot specify the parameters in the Request Parameter text box, so instead, after adding the URL Sequence, go to corresponding URL's 'Edit Monitor' page and enter the parameters. Consider the following URL Sequence :

• URL 1 : https://api.appmanager.com/oauth/token?client_id=32faff282a68a03c2eb0f41aeca63a88&grant_type=client_credentials&client_secret=9de7337627ab85f3
• URL 2 : https://api.appmanager.com/bestoffer/97232/354992/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384/li>
• URL 3 : https://api.appmanager.com/selectoffer/97232/3549/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384
• ...
• ...
• URL N : ...

In the above example access_token is a dynamic parameter whose value changes every time we login. In order to monitor the above sequence, follow the below mentioned steps :

1. Download the Transaction Recorder tool and record a new transaction sequence.
2. The recorder tool will have recorded all the steps along with dynamic request parameter for that session.
3. After adding the URL Sequence, click 'Edit Monitor' for the corresponding URL to enter the parameters.
4. In the Edit Monitor page, for the field 'Request Parameters (if any)' provide the dynamic request parameter as $parameter=TEXT:-:<parameter_name>:-:

   For Eg - In URL 1: The 'parameter' is 'access_token' and 'parameter_name' is 'access_token='. So, for the text box correponding to the 'Request Parameters' provide $access_token=TEXT:-:access_token=:-: and save the page

5. Click 'Edit Monitor' for the second URL, remove the dynamic request parameter from the 'URL address' field and use the same parameter to fill the 'Request Parameters (if any)' field

   For Eg - In URL 2: Edit https://api.appmanager.com/bestoffer/97232/354992/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384, remove the dynamic request parameter (access_token) and change the 'URL address' to https://api.appmanager.com/bestoffer/97232/354992/offers.xml?. Provide the 'Request Parameters' as $access_token=fcdee0e94da9b934a4cef6c2dee70384

6. Repeat the above steps for the rest of the URLs in the sequence where the dynamic parameter is present

So for every polling interval Applications Manager will search for this dynamically generated parameter and it will replace it.

Applications Manager acts as a continuous URL monitoring service that keeps a constant watch over the specified URL or web site pages. URL monitors verify the availability of specified, addressable, standard HTTP and HTTPS URLs. They scan the HTTP and HTTPS pages looking for a predefined keyword to check whether the web site is available.

There are two ways of URL monitoring provided by Applications Manager.

1. URL Monitoring
2. URL Sequence (Record & Playback)

In URL Sequence, click on the individual URL listed, to view its monitored parameters.

Monitored Parameters

• Availability tab gives the Availability history for the past 24 hours or 30 days.
• Performance tab gives the Health Status and events for the past 24 hours or 30 days.
• List view enables you to perform bulk admin configurations.

Click on the individual monitors listed, to view the following information.