This article assumes you already have Visual Studio installed (I’m using VS 2017), an Azure account, and a queue ready to be monitored.
I have these web jobs that are asynchronously doing all this work, pulling data off queues, but I don’t know when they are really “done.”
So, we’ve got this awesome website, hosted on Azure, that pulls URLs from SQL server and needs to do some web scraping. After the pages are scraped, the site needs to analyze the content and do some more magic. To do this on a grand scale, we’re leveraging Azure queues and scalable instances of Web Jobs to crank through the work. As the data is processed, the results are stored into table storage. Eventually, the results will be exported in JSON format.
Unsurprisingly, the client wants this end-to-end processing to happen as quickly as possible. In addition, we want to know the queue is empty as soon as possible so we can perform the final step—the export to JSON. We could open up the Azure Storage Explorer, find the queue, and keep hitting refresh. But why would we do that when we have computers to do the work?
We want the solution to:
Leverage the same website
Bonus Points!
When the queue is empty:
Let’s first create a brand new, C# Console Application called QueueWatcher.
Windows.AzureStorage (and all the accoutrements):
Microsoft.WindowsAzure.ConfigurationManager
<?xml version="1.0" encoding="utf-8" ?> <configuration> <startup> <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" /> </startup> <appSettings> <add key="StorageConnectionString" value="DefaultEndpointsProtocol=s;AccountName=[YourAccountName];AccountKey=[YourAccountKey];EndpointSuffix=core.windows.net" /> </appSettings> </configuration>
An instance of the CloudQueueWatcher class (explained below) is created. An asynchronous task is started that periodically invokes the OnTick function. The OnTick function calls the CheckQueueCount function and writes the result to the console.
using Microsoft.Azure; using Microsoft.WindowsAzure.Storage; using Microsoft.WindowsAzure.Storage.Queue; using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading; using System.Threading.Tasks;
namespace QueueWatcher { class Program { private static CloudQueueWatcher cqw; static void Main( string [ ] args ) { try { string queueName = "urlqueue"; cqw = new CloudQueueWatcher( queueName ); // Make sure the queue exists if ( cqw.Queue == null ) { Console.WriteLine( "Queue {0} not found", queueName ); return; } else { Console.WriteLine( "Queue {0} found!", queueName ); } char input = '\0'; Console.WriteLine( "Enter q to quit" ); Task t = RunPeriodicAsync( OnTick, TimeSpan.FromSeconds( 5 ), TimeSpan.FromSeconds( 5 ), CancellationToken.None ); while ( input != 'q' ) { t.ConfigureAwait( false ); input = Console.ReadKey().KeyChar; } } catch ( Exception ex ) { Console.WriteLine( "An error occurred: {0}", ex.Message ); } finally { Console.ReadLine(); } } // The 'onTick' method will be called periodically unless cancelled. private static async Task RunPeriodicAsync( Action onTick, TimeSpan dueTime, TimeSpan interval, CancellationToken token ) { // Initial wait time before we begin the periodic loop. if ( dueTime > TimeSpan.Zero ) await Task.Delay( dueTime, token ); // Repeat this loop until cancelled. while ( ! token.IsCancellationRequested ) { // Call our onTick function. onTick?.Invoke(); // Wait to repeat again. if ( interval > TimeSpan.Zero ) await Task.Delay( interval, token ); } } private static void OnTick() { Console.WriteLine( "Queue Count = {0}", cqw.CheckQueueCount() ); } } }
This creates a reference to the Azure Storage Account, a client to interact with the account, and a reference to the queue we want to track. The FetchAttributes method will grab info from the queue, including the approximate count.
using Microsoft.Azure; using Microsoft.WindowsAzure.Storage; using Microsoft.WindowsAzure.Storage.Queue; using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace QueueWatcher { public class CloudQueueWatcher { public CloudStorageAccount StorageAccount { get; set; } public CloudQueueClient QueueClient { get; set; } public CloudQueue Queue { get; set; } public CloudQueueWatcher() { } public CloudQueueWatcher( string queueName ) { StorageAccount = CloudStorageAccount.Parse( CloudConfigurationManager.GetSetting("StorageConnectionString" ) ); // Create the queue client. CloudQueueClient queueClient = StorageAccount.CreateCloudQueueClient(); // Retrieve a reference to a queue. Queue = queueClient.GetQueueReference( queueName ); } public int CheckQueueCount() { Queue.FetchAttributes(); return Queue.ApproximateMessageCount ?? 0; } } }
To test, a web application loads the queue, and an Azure function takes messages off the queue. Now I can run the CQW to track progress.
That’s all there is to it!
For my next trick, I will investigate enhancing CQW to write/update SQL and send notifications when the queue is empty. And If your organization needs help with any Azure issue, please don’t hesitate to reach out to Anexinet. We’d love to help you clear things up.
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
Cookie | Duration | Description |
---|---|---|
cookielawinfo-checbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.