Efficiently Handling Long-Running Tasks in Web Applications with Background Jobs and Real-Time Updates

In web application development, you often encounter tasks that take a significant amount of time to complete. Executing these long-running operations directly within a standard web request can lead to several issues, including:

• Request Timeouts: Web servers often have timeout limits. If a task takes longer than this limit, the connection can be terminated, leaving the user with an error and incomplete work.
• Memory Spikes: Long processes, especially those involving large datasets, can consume excessive memory, potentially impacting server performance and stability.
• Poor User Experience: Users are left waiting without feedback. A stalled request makes the application feel unresponsive and frustrating.

The solution? Offload these tasks to background jobs, processed outside the normal request-response cycle. This frees up the web server to handle other requests quickly, improving responsiveness and user experience.

The Power of Background Jobs

Background jobs are tasks executed asynchronously, separate from the user’s direct interaction. This approach offers significant advantages:

1. Improved Responsiveness: The web server immediately responds to the user, acknowledging that the task has been accepted. The actual processing happens in the background.
2. Enhanced Scalability: By distributing tasks across multiple workers, your application can handle a higher volume of long-running operations without overwhelming the main web server.
3. Resource Management: Background jobs can be configured with resource limits (memory, CPU) to prevent runaway processes from impacting system stability.
4. Retry Mechanisms: If a background job fails, it can be automatically retried, providing resilience against temporary errors.

But how do we keep the user informed? That’s where real-time communication comes in.

Real-Time Progress Updates with WebSockets

When a task moves to the background, the user loses visibility into its progress. WebSockets provide a solution: a persistent, bidirectional communication channel between the server and the client (browser). This allows the background job to send real-time updates to the user without the need for constant polling.

Benefits of WebSockets for Background Job Updates:

• Instant Feedback: Users receive immediate updates on task progress, eliminating uncertainty.
• Efficient Communication: Unlike traditional polling, WebSockets maintain an open connection, reducing overhead and latency.
• Enhanced User Experience: Real-time updates provide a more engaging and informative experience.

A Practical Example: Exporting Large CSV Files

Let’s consider a common scenario: exporting a large dataset to a CSV file. This operation can be time-consuming, especially with numerous records. Here’s how to implement it efficiently using background jobs and WebSockets:

1. Create a Background Job:

   Instead of performing the CSV export directly in the controller, we create a dedicated background job (e.g., ExportDataJob). This job encapsulates the logic for fetching the data, generating the CSV, and saving it to a temporary location.

   # app/jobs/export_data_job.rb
   require 'csv'
   
   class ExportDataJob < ApplicationJob
     queue_as :default
   
     def perform(data_type)
       # 1. Fetch the data based on data_type
       data = fetch_data(data_type)
   
       # 2. Generate the CSV file
       filename = Rails.root.join('tmp', "#{data_type}_export_#{SecureRandom.uuid}.csv")
       CSV.open(filename, 'w') do |csv|
         # Add header row
         csv << data.first.keys
   
         # Add data rows, providing progress updates
         data.each_with_index do |row, index|
           csv << row.values
           progress = ((index + 1).to_f / data.size * 100).round
           broadcast_progress(progress, data_type)
         end
       end
   
       #3. Notify client.
         broadcast_completion(filename, data_type)
     end
   
   private
   
      def fetch_data(data_type)
       # Replace with logic to fetch the actual data (e.g., from a database)
        case data_type
        when 'photos'
           Photo.all.map { |photo| { id: photo.id, title: photo.title, url: photo.url } }
        # Add more cases as needed for different data types
       else
         []
        end
     end
   
     def broadcast_progress(progress, data_type)
       ActionCable.server.broadcast(
         "export_#{data_type}_channel", # Unique channel for each data type
         { progress: progress }
       )
     end
   
      def broadcast_completion(filename, data_type)
       ActionCable.server.broadcast(
         "export_#{data_type}_channel",
         {  file_url: "/downloads/#{File.basename(filename)}" }
       )
     end
   end
   
   

2. Trigger the Job from the Controller:

   The controller initiates the background job and immediately responds to the user.

   # app/controllers/exports_controller.rb
   class ExportsController < ApplicationController
     def create
       # Get the type of data to export (e.g., from params[:data_type])
       data_type = params[:data_type] || 'photos' # Default to 'photos'
   
       # Enqueue the background job
       ExportDataJob.perform_later(data_type)
   
       # Respond immediately to the user
       render json: { message: 'Export started. You will be notified when it is complete.' }, status: :accepted
     end
   
     def download
        filename = params[:filename]
        filepath = Rails.root.join('tmp', filename)
        if File.exist?(filepath)
           send_file filepath, filename: filename, type: 'text/csv'
         else
             head :not_found
         end
      end
   end
   

3. Set Up a WebSocket Channel:

   Create an Action Cable channel to handle real-time communication.

   # app/channels/export_channel.rb
   class ExportChannel < ApplicationCable::Channel
     def subscribed
       # Use a dynamic stream name based on the data type being exported.
       stream_from "export_#{params[:data_type]}_channel"
     end
   
     def unsubscribed
       # Any cleanup needed when channel is unsubscribed
     end
   end
   

4. Client-Side Handling (JavaScript):

   The client-side JavaScript code establishes a WebSocket connection, listens for updates, and displays the progress to the user.

   // app/javascript/controllers/export_controller.js
   import { Controller } from "@hotwired/stimulus"
   import { createConsumer } from "@rails/actioncable"
   
   export default class extends Controller {
     static targets = ["progress", "progressBar", "status"]
   
     connect() {
       this.consumer = createConsumer();
       this.channel = null;
       this.dataType = this.data.get("type") || 'photos'; // Get data type, default to 'photos'
     }
   
     startExport() {
          fetch(`/exports?data_type=${this.dataType}`, { method: 'POST', headers: {'X-CSRF-Token': document.querySelector('meta[name="csrf-token"]').content}})
          .then(response => response.json())
         .then(data => {
          this.statusTarget.textContent = data.message; // Update with message from server
          this.subscribeToChannel();
          })
       .catch(error => {
       console.error("Error starting export:", error);
       this.statusTarget.textContent = "Error starting export.";
        });
     }
   
     subscribeToChannel() {
         if (this.channel) {
             return; // Prevent multiple subscriptions
          }
         this.channel = this.consumer.subscriptions.create(
            { channel: "ExportChannel", data_type: this.dataType },
           {
             connected: () => {
              this.statusTarget.textContent = "Export in progress...";
             },
           received: (data) => {
               if (data.progress !== undefined) {
                  this.progressBarTarget.style.width = `${data.progress}%`;
                  this.progressTarget.textContent = `${data.progress}%`;
                } else if (data.file_url) {
                     this.statusTarget.textContent = "Export complete!";
                   window.location.href = data.file_url; // Redirect to download
               }
          },
          disconnected: () => {
              this.statusTarget.textContent = "Export channel disconnected.";
              this.channel = null;
               }
         }
       );
     }
   
     disconnect() {
         if (this.channel) {
              this.channel.unsubscribe();
              this.channel = null;
         }
      }
   }
   

    <!--app/views/your_view.html.erb -->
   <div data-controller="export" data-export-type="photos">
   
       <button data-action="click->export#startExport">Export Photos</button>
       <p data-export-target="status"></p>
   
       <div class="progress-wrapper" >
          <div class="progress-bar" data-export-target="progressBar" style="width: 0%;"></div>
       </div>
       <div data-export-target="progress">0%</div>
   </div>
   
   

* Added routes.rb to define the download

   post '/exports', to: 'exports#create'
   get '/downloads/:filename', to: 'exports#download', as: 'download'

Key Improvements and Explanations:

• Dynamic Channels: The ExportChannel now uses stream_from "export_#{params[:data_type]}_channel". This creates a unique channel for each type of data being exported (e.g., “export_photos_channel”, “export_users_channel”). This is crucial to prevent updates for one export type from interfering with another. A user exporting photos shouldn’t see progress updates from a user exporting users.
• Data Type Handling: The ExportDataJob now accepts a data_type argument. This makes the job reusable for different export types. The fetch_data method is a placeholder; you’d replace its contents with your actual data retrieval logic (e.g., querying your database models). The controller sets this type.
• Client-Side Data Type: The Stimulus controller (export_controller.js) now retrieves the data_type from a data attribute on the HTML element (data-export-type). This tells the client-side code which channel to subscribe to.
• Clearer Status Updates: The client-side JavaScript updates a statusTarget element with messages, providing more informative feedback to the user (e.g., “Export started,” “Export in progress,” “Export complete”).
• Progress Bar: The example includes a basic progress bar (progress-bar) and a text display of the percentage (progressTarget).
• Direct Download: The broadcast_completion method inside ExportPhotosJob now uses file_url and redirect to the download link , also the download method added to the controller to handle serving the file .
• Error Handling: The JavaScript includes basic error handling to catch issues during the export process.
• CSRF Token: to send the request successfully.
• Prevent Multiple Subscriptions the code has been edited to prevent multiple subscription for the channel.
• Disconnect Channel: Added disconnect method to unsubscribe from channel when it is not needed.

This complete example demonstrates the core principles of using background jobs and WebSockets for long-running tasks. You can adapt this approach to various scenarios, providing a significantly better user experience for your web applications.

How Innovative Software Technology Can Help

At Innovative Software Technology, we specialize in building robust and scalable web applications. We have extensive experience with background processing and real-time communication technologies, including:

• Background Job Frameworks: We are proficient in using frameworks like Sidekiq, Resque, and Solid Queue to manage background tasks efficiently.
• WebSocket Technologies: We leverage Action Cable, Socket.IO, and other WebSocket solutions to implement real-time features.
• Performance Optimization: We optimize applications for speed and scalability, ensuring they can handle heavy workloads and deliver a smooth user experience.
• Custom Solutions: We build custom solutions for progress tracking, notifications, and other real-time interactions based on your application’s specific requirements.

If you’re facing challenges with long-running tasks, slow response times, or a need for real-time updates in your web application, contact Innovative Software Technology. We can help you design and implement a solution that improves performance, enhances user experience, and ensures the reliability of your application.