Google Search Results in Node.js

SerpApi client library for Node.js. Previously: Google Search Results Node.js.

Description

This NodeJS module is designed to scrape and parse Google, Google Scholar, Google Maps, Bing, Baidu, Yandex, Yahoo, eBay, Walmart, Home Depot, and other search engine results using SerpApi service.

Access structured JSON results in real-time without the hassle of dealing with proxies and CAPTCHAs.

Programming language: JavaScript

License: MIT License

Tags: API     Crawler     Third-party APIs     Crawl     Scraping     API Wrapper    

README

Google Search Node.js

This NodeJS module is designed to scrape and parse Google, Bing and Baidu results using SerpApi. This Ruby Gem is meant to scrape and parse Google results using SerpApi. The following services are provided:

• Search API
• Location API
• Search Archive API
• Account API

SerpApi provides a script builder to get you started quickly.

This npm package is meant to scrape and parse Google results using SerpApi. The following services are provided:

• Search API
• Location API
• Search Archive API
• Account API

SerpApi provides a script builder to get you started quickly.

The full documentation is available here.

Link to NodeJS Package

Requirement

• ES6 basic understanding
• NodeJS coding skills
• Node 7+ and NPM installed

Installation

NPM 7+

$ npm install google-search-results-nodejs

Link to npm package

Quick start

const SerpApi = require('google-search-results-nodejs')
const search = new SerpApi.GoogleSearch("Your Private Key")
search.json({
 q: "Coffee", 
 location: "Austin, TX"
}, (result) => {
  console.log(result)
})

This example runs a search about "coffee" using your secret api key.

The SerpApi service (backend)

• searches on Google using the search: q = "coffee"
• parses the messy HTML responses
• return a standardizes JSON response The class GoogleSearch
• Format the request to SerpApi server
• Execute GET http request
• Parse JSON into Ruby Hash using JSON standard library provided by Ruby Et voila..

Alternatively, you can search:

• Bing using BingSearch class
• Baidu using BaiduSearch class
• Yandex using YandexSearch class
• Ebay using EbaySearch class
• Yahoo using YahooSearch class

See the playground to generate your code.

Example

• How to set SERP API key
• Search API capability
• Example by specification
• Location API
• Search Archive API
• Account API
• Promise and callback
• Coding style

How to set SERP API key

The SerpApi api_key can be set globally using a singleton pattern.

const SerpApi = require('google-search-results-nodejs')
let search = new SerpApi.GoogleSearch("Your Private Key")

The SerpApi api_key can be provided for each request

const SerpApi = require('google-search-results-nodejs')
let search = new SerpApi.GoogleSearch()
let result = search.json({
 api_key: "Your private key",
 q: "Coffee",            // search query
 location: "Austin, TX", // location 
}, (data) => {
  console.log(data)
})

Search API capability

query_params = {
  q: "query",
  google_domain: "Google Domain", 
  location: "Location Requested", 
  device: device,
  hl: "Google UI Language",
  gl: "Google Country",
  safe: "Safe Search Flag",
  num: "Number of Results",
  start: "Pagination Offset",
  api_key: "Your SERP API Key",  // https://serpapi.com/dashboard
  tbm: "nws|isch|shop",
  tbs: "custom to be search criteria",
  async: true|false,   // allow async query
  output: "json|html", // output format
}

const SerpApi = require('google-search-results-nodejs')
const search = new SerpApi.GoogleSearch()

// create a callback
callback = (data) => {
 console.log(data)
}

// Show result as JSON
search.json(query_params, callback)

// Show result as HTML file
search.html(query_params, callback)

the full documentation

see below for more hands on examples.

Example by specification

We love true open source, continuous integration and Test Drive Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality Of Service).

The directory test/ includes specification/examples.

Set your api key.

export API_KEY="your secret key"

Run all tests npm test

Location API

const search = new SerpApi.GoogleSearch(api_key)
search.location("Austin", 3, (data) => {
  console.log(data)
})

it prints the first 3 location matching Austin (Texas, Texas, Rochester)

[ { id: '585069bdee19ad271e9bc072',
    google_id: 200635,
    google_parent_id: 21176,
    name: 'Austin, TX',
    canonical_name: 'Austin,TX,Texas,United States',
    country_code: 'US',
    target_type: 'DMA Region',
    reach: 5560000,
    gps: [ -97.7430608, 30.267153 ],
    keys: [ 'austin', 'tx', 'texas', 'united', 'states' ] },
  ...]

Search Archive API

The first search result returns a search_id which can be provided to get the search result from the archive.

var search = new SerpApi.GoogleSearch(api_key)
search.json({q: "Coffee", location: "Portland" }, (search_result) => {
  // search in archive for the search just returned
  search.search_archive(search_result.search_metadata.id, (archived_search) => {
    console.log(archived_search)
  })
})

it prints the search from the archive.

Account API

const search = new SerpApi.GoogleSearch(api_key)
search.account((data) => {
  console.log(data)
})

it prints your account information.

Promise and callback

This API was developped using basic callback to handle response. And exception are just throw away with interceptor.

if you want to take advantage of the promise to block the request. here is how I will do.

const util = require('util')

function getJson(parameter, resolve, reject) {  
  const search = new SerpApi.GoogleSearch(api_key)
  try {
    search.json(parameter, resolve)
  } catch (e) {
    reject(e)
  }
}

const blockFn = util.promisify(getJson)
blockFn[util.promisify.custom](parameter).then((data) => {
  expect(data.local_results[0].title.length).toBeGreaterThan(5)
  done()
}).catch((error) => {
  console.error(error)
  done()
})

Reference:

• test: test/ExampleSpec.js
• documentation: https://nodejs.org/docs/latest-v8.x/api/util.html#util_util_promisify_original

Coding style

This API is using callback to run in non-blocking code. Here we are trying to follow the spirit of NodeJS. For reference you can read this article:

• https://nodejs.org/en/docs/guides/blocking-vs-non-blocking/
• https://nodejs.org/en/docs/guides/dont-block-the-event-loop/

For pratical example, you can see the test located under test/.

Run regression

To run the regression suite.

export API_KEY="your api key"
make test

Change log

• 2.0.1
  • fix classes loading.
• 2.0
  • Refractor class name: SearchResult -> Search
• 1.2
  • stable version to support all the basic search API.

Conclusion

SerpApi supports Google Images, News, Shopping and more.. To enable a type of search, the field tbm (to be matched) must be set to:

• isch: Google Images API.
• nws: Google News API.
• shop: Google Shopping API.
• any other Google service should work out of the box.
• (no tbm parameter): regular Google search.

The field tbs allows to customize the search even more.

The full documentation is available here.