Taleh Rzayev
All projects

Hobby2025

Vacancy Scraper Bot

A Telegram bot that collects jobs and internships from Azerbaijani job sites and publishes them to channels.

Source code
Sources
HelloJob.az · Tecrube.az
Commands
7
Channels
Test + Prod
Database
MongoDB

01Overview

The bot replaces the manual work of copying vacancies into Telegram channels. Admins trigger scraping, review what was collected, and publish in batches - first to a test channel, then to production.

02Architecture

A thin command layer sits on top of source-specific scraper services and Mongoose models. Every command passes through an admin-only middleware.

Telegram

Admin chat

/scrap /send /list /log

Channels

Test & production

Bot

Auth middleware

ALLOWED_USERS whitelist

Command router

One module per command

Message handler

Per-source formatting

Scrapers

Source 1

HelloJob.az

Source 2

Tecrube.az

Toolkit

Axios + Cheerio, Puppeteer

Data

Job

Vacancies

Intern

Internships

ScrapeInfo

Last run per source

03Components

Scraper services

Extract listings

  • Axios + Cheerio for static HTML, Puppeteer for pages that render with JavaScript.
  • Each source returns the same shape: title, link, company, date and description.

Publishing

Controlled delivery

  • /send posts the next 10 unsent listings, or one specific post by id.
  • Message formats differ per source (e.g. start and end dates for internships).

04How it works

  1. 1

    /scrap

    Admin picks a source; the scraper fetches and parses the listing page.

  2. 2

    Deduplicate

    Each listing is looked up by link; only new ones are stored.

  3. 3

    Record

    ScrapeInfo stores the time of the last run for /start statistics.

  4. 4

    /send test

    A batch goes to the test channel for review.

  5. 5

    /send prod

    The same batch is published; /log shows what was sent.

05Technical deep dives

Idempotent scraping

The link is the natural key of a vacancy. Re-running a scrape never creates duplicates, so admins can scrape as often as they like.

services/scraper_model_1.jsjs
let addedCount = 0;
for (const job of jobs) {
  const exists = await Job.findOne({ link: job.link });
  if (!exists) {
    await Job.create(job);
    addedCount++;
  }
}

await ScrapeInfo.findOneAndUpdate(
  { sourceId: 1 },
  { lastScrapedAt: new Date() },
  { upsert: true, new: true },
);

06Design decisions

Admin-only by design

A whitelist middleware guards every command, so the bot can live in public channels safely.

Sources as configuration

Sources are registered through environment config, so adding a site means adding one scraper module.

Test before prod

Two channels make publishing reviewable without a separate admin UI.

07Tech stack

Runtime
Node.jsExpressnode-telegram-bot-api
Scraping
AxiosCheerioPuppeteer
Data
MongoDBMongoose

Next case study

Aments Storefront

A server-rendered Next.js storefront with a strict Route → Query → Mapper → UI data architecture.