Ever wondered how much artists truly earn per stream on Spotify? That burning question led me to embark on a fascinating personal project called RevenueSpot. This endeavor was a deep dive into web scraping and the intricacies of the Spotify API, born from a simple curiosity about the financial side of music streaming.

The Quest for Spotify Earnings

The spark for RevenueSpot ignited during the infamous Drake-Kendrick Lamar feud. My friends and I, musing about the massive stream counts, wondered about the actual revenue generated for these artists. The widely cited figure of roughly $0.00028 per stream fueled my desire to put this to the test. My initial thought was to leverage the Spotify API directly for play counts. However, a quick exploration revealed a surprising limitation: the Spotify API, despite its rich data, doesn’t provide direct access to track play counts – a highly dynamic metric. This unexpected hurdle pushed me towards a more unconventional solution.

Mastering Web Scraping for Data

With the API route blocked, web scraping became the essential path forward. My professional background had provided me with prior experience in web scraping, notably transitioning from Java with Selenium to Python with BeautifulSoup for greater efficiency. For RevenueSpot, however, I found myself in a new Node.js environment, leading me to adopt Puppeteer – a powerful library for headless browser automation.

The journey wasn’t without its challenges. Extracting the precise play count element from Spotify’s dynamic web pages required careful inspection and a bit of trial and error to find the unique HTML selectors. More significantly, I encountered the common adversary of web scrapers: bot detection. Overcoming this involved strategic implementation of timeouts, careful selection of Chromium browser versions, and fine-tuning HTTP request configurations to mimic human interaction and bypass security measures. After diligent effort, RevenueSpot successfully extracted the elusive stream data.

From Code to Cloud: Deployment Challenges

With the core functionality working, the next step was deployment. I quickly spun up an AWS EC2 instance, a familiar environment from past projects, to host the application. The scraping process, which involved launching a headless browser and waiting for page data, necessitated a persistent environment. While researching deployment, AI tools suggested AWS Elastic Beanstalk as a cost-effective solution for projects with intermittent usage. This was the optimal choice, yet, due to a tight schedule and a desire to avoid a new learning curve, I opted for the simpler, familiar EC2 setup.

This decision, unfortunately, proved to be a costly lesson. A recurring $10 monthly charge from AWS, entirely unexpected for a personal project not actively shared or promoted, quickly accumulated over several months. This oversight highlighted the critical importance of careful infrastructure planning and understanding cloud billing, especially for hobby projects.

Key Takeaways from RevenueSpot

While RevenueSpot might not have become a widely adopted tool, the entire process offered invaluable insights. Beyond the technical triumphs of web scraping with Puppeteer and navigating API limitations, the project underscored vital lessons in project management and cloud economics. It taught me the importance of thorough upfront planning for infrastructure, even for small-scale endeavors, and the potential pitfalls of defaulting to familiar solutions without fully considering cost-efficiency. It was a stark reminder to approach impulsive project ideas with more comprehensive consideration, from conception to deployment.

The End of a Chapter

Eventually, after realizing the accumulating costs and shifting focus to new ventures, I decommissioned the RevenueSpot EC2 instance. The project served its purpose: a fantastic learning experience in web development, scraping, and the often-unseen costs of cloud services.