Transforming Web Search Results into Markdown: A No-Code Approach for Enhanced Data Processing

In today’s digital landscape, extracting and structuring information from the web is crucial for various applications, including feeding data into Large Language Models (LLMs). This post explores a streamlined, no-code method to convert Google Search results into Markdown format, a human-readable and easily parsed text format ideal for LLMs and other text-processing tasks. We’ll leverage powerful online tools to achieve this, making the process accessible even without extensive coding knowledge.

Why Markdown?

Markdown is a lightweight markup language that uses plain text formatting syntax. Its simplicity and readability make it an excellent choice for:

Content Creation: Easily format text for blogs, documentation, and notes.
LLM Input: Provide clean, structured text input to LLMs for improved processing and understanding.
Data Portability: Markdown is platform-independent and can be easily converted to other formats like HTML.

The No-Code Solution: Combining Search APIs and Content Extraction

This method utilizes two key services:

SerpApi: A powerful API that retrieves structured data from various search engines, including Google. It handles the complexities of web scraping, allowing you to focus on the data itself.
Jina AI’s Reader API: This service specializes in extracting the main content from web pages and converting it into structured formats, including Markdown. It preserves the essential document structure, making the output clean and organized. Jina AI offers a generous free tier with 1 million test credits to get started, although keep in mind it’s usage based, not request based.

Method 1: Using Make (Formerly Integromat)

Make is a visual platform for connecting apps and automating workflows. SerpApi provides a dedicated Make app, simplifying the integration process. Here’s how to build the workflow:

Create a New Scenario: Start by creating a new scenario in Make.
Add the SerpApi App: Search for and add the SerpApi app. Select the “Google Search” module.
Configure SerpApi:
- Create a connection by entering your SerpApi API key (available on your SerpApi dashboard).
- Set your search query, location, and other desired parameters.
- You can increase the “Pagination Limit” to fetch more results, but remember that each page consumes one SerpApi search credit.
Test the SerpApi Module: Run the module once to verify that it returns the expected search results.
Add an Iterator Module: Insert an “Iterator” module after SerpApi. This module will process each search result (specifically, each item in the organic_results array) individually.
Add an HTTP “Make a Request” Module: This module will send requests to the Jina AI Reader API.
- URL: Set the URL to `https://r.jina.ai/` followed by the “link” attribute from the Iterator module (this dynamically inserts the URL of each search result).
- Headers: Add an Authorization header with the value Bearer YOUR_JINA_AI_READER_API_KEY (replace with your actual API key).
Add a Google Docs “Create a Document” Module (Optional): This step demonstrates how to store the extracted Markdown content. You can replace this with any other integration that suits your needs (e.g., sending the data to an AI platform, a database, etc.).
- Connect your Google account.
- Set the document name and pass the content from the Reader API’s output to the document’s content field.
Testing and Limiting (Optional): During testing, add a filter between the Iterator and HTTP modules. Set a condition to continue only if the Iterator’s “position” attribute is equal to “1”. This limits the workflow to processing only the first result, saving API credits during development. Remove this limiter before deploying the full workflow.

Method 2: Using Zapier

Zapier is another popular automation platform. While SerpApi doesn’t have a native Zapier integration, you can still achieve the same result using Zapier’s “Code by Zapier” action. This requires a bit of JavaScript, but we’ll provide the code and explain each step. Keep in mind that the Free plan has a 1 second code runtime limit.

Create a New Zap: Start by creating a new Zap.
Add a Trigger: Choose a trigger that suits your needs. For example, you could use the “Schedule by Zapier” trigger to run the Zap at a specific time.

Add a “Code by Zapier” Action (SerpApi Request):

Language: Select JavaScript.
Input Data:
- SERPAPI_KEY: Your SerpApi API key.
- QUERY: Your search query.
- LOCATION: Your desired search location.
- Add other serapi parameters if you desire.
Code: Paste the following JavaScript code:

const { SERPAPI_KEY, QUERY, LOCATION } = inputData;

async function fetchGoogleSearchResults(query, location) {
  try {
    const response = await fetch(
      `https://serpapi.com/search.json?engine=google&q=${encodeURIComponent(
        query
      )}&location=${encodeURIComponent(
        location
      )}&gl=us&hl=en&api_key=${SERPAPI_KEY}`
    );

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (!data) {
      throw new Error("No data returned from SerpAPI");
    }

    return data;
  } catch (error) {
    throw new Error(`SerpAPI request failed: ${error.message}`);
  }
}

try {
  const searchResults = await fetchGoogleSearchResults(QUERY, LOCATION);

  output = {
    organicResults: searchResults?.organic_results || "No Organic Result Found"
  };

  return output;
} catch (error) {
  console.error("Error:", error.message);
  return { organic_results: [] };
}

Test the Code Step: Ensure the code runs correctly and retrieves search results from SerpApi.
Add a “Looping by Zapier” Action:
- Values to Loop: Select the “Organic Results Link” from the previous Code step’s output. Assign it to a key (e.g., “links”).
- Maximum Iterations (for testing): Set this to 1 to process only one result during testing.

Add a “Code by Zapier” Action (Reader API Request) Inside the Loop:

Language: JavaScript.
Input Data:
- link: The “link” value from the Looping action.
- READERAPI_KEY: Your Jina AI Reader API key.
Code: Paste the following JavaScript code:

const { link, READERAPI_KEY } = inputData;

async function fetchReaderData(link, apiKey) {
  try {
    const encodedUrl = encodeURIComponent(link);

    const response = await fetch(`https://r.jina.ai/${encodedUrl}`, {
      method: 'GET',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Accept': 'application/json'
      }
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (!data) {
      throw new Error('No data returned from Reader API');
    }

    return data;
  } catch (error) {
    console.error('Reader API request failed:', error.message);
    throw error;
  }
}

try {
  const readerResults = await fetchReaderData(link, READERAPI_KEY);

  output = {
    readerResults
  };

  return output;
} catch (error) {
  return {
    error: error.message,
    content: null,
    title: null,
    metadata: null
  };
}

Test the Code Step: Verify that the code retrieves data from the Reader API.
Add a “Create Document From Text in Google Docs” Action (Optional) Inside the Loop:
- Connect your Google account.
- Configure the action:
  - Document Name: Choose a name for your document (you can use dynamic values from previous steps).
  - Folder: Select the Google Drive folder where you want to save the document.
  - Content: Select the “content” field from the Reader API output (from the second Code step).
Test the Entire Zap: Run a complete test to ensure all steps work together seamlessly. Check your Google Drive for the generated document(s).

Conclusion

No-code platforms like Make and Zapier, combined with powerful APIs like SerpApi and Jina AI’s Reader API, empower users to extract and transform web data without writing complex code. This opens up possibilities for various applications, from feeding structured data to LLMs to automating content creation and research workflows.

Innovative Software Technology specializes in creating custom software solutions that leverage cutting-edge technologies, including AI and data processing. If you need to build sophisticated workflows, integrate with specific APIs, or develop custom data extraction and transformation tools, we can help. Our expertise in no-code/low-code development, API integration, and AI-powered solutions allows us to deliver efficient and scalable solutions tailored to your unique requirements. Contact us today to discuss your project and explore how we can transform your data processing needs into reality.