If you need to extract metadata such as title, description, keywords, Open Graph tags, canonical URL, and Twitter Card information from an HTML page, Python makes this easy using requests and BeautifulSoup.

Install Required Packages

Install the required Python packages:

pip install requests beautifulsoup4

Basic HTML Metadata Scraper

Create a Python file named metadata.py:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"

response = requests.get(
    url,
    headers={
        "User-Agent": "Mozilla/5.0"
    },
    timeout=10
)

response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

title = soup.title.string.strip() if soup.title and soup.title.string else None

description = soup.find(
    "meta",
    attrs={"name": "description"}
)

keywords = soup.find(
    "meta",
    attrs={"name": "keywords"}
)

print("Title:", title)
print("Description:", description.get("content") if description else None)
print("Keywords:", keywords.get("content") if keywords else None)

Run it:

python metadata.py

Example output:

Title: Example Domain
Description: This domain is for use in illustrative examples.
Keywords: example, domain

Extract Open Graph Metadata

Many modern websites use Open Graph metadata for sharing pages on Facebook, LinkedIn, WhatsApp, and other platforms.

A typical HTML page contains:

<meta property="og:title" content="My Website" />
<meta property="og:description" content="My website description" />
<meta property="og:image" content="https://example.com/image.jpg" />
<meta property="og:url" content="https://example.com/" />

You can extract these tags with:

og_title = soup.find(
    "meta",
    attrs={"property": "og:title"}
)

og_description = soup.find(
    "meta",
    attrs={"property": "og:description"}
)

og_image = soup.find(
    "meta",
    attrs={"property": "og:image"}
)

og_url = soup.find(
    "meta",
    attrs={"property": "og:url"}
)

print("OG Title:", og_title.get("content") if og_title else None)
print("OG Description:", og_description.get("content") if og_description else None)
print("OG Image:", og_image.get("content") if og_image else None)
print("OG URL:", og_url.get("content") if og_url else None)

Extract Twitter Card Metadata

Twitter/X uses Twitter Card metadata to control how a URL appears when shared.

For example:

<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="My Website" />
<meta name="twitter:description" content="My website description" />
<meta name="twitter:image" content="https://example.com/image.jpg" />

Extract them using:

twitter_card = soup.find(
    "meta",
    attrs={"name": "twitter:card"}
)

twitter_title = soup.find(
    "meta",
    attrs={"name": "twitter:title"}
)

twitter_description = soup.find(
    "meta",
    attrs={"name": "twitter:description"}
)

twitter_image = soup.find(
    "meta",
    attrs={"name": "twitter:image"}
)

print("Twitter Card:", twitter_card.get("content") if twitter_card else None)
print("Twitter Title:", twitter_title.get("content") if twitter_title else None)
print("Twitter Description:", twitter_description.get("content") if twitter_description else None)
print("Twitter Image:", twitter_image.get("content") if twitter_image else None)

Extract Canonical URL

The canonical URL is usually defined as:

<link rel="canonical" href="https://example.com/page" />

Extract it with:

canonical = soup.find(
    "link",
    attrs={"rel": "canonical"}
)

print(
    "Canonical:",
    canonical.get("href") if canonical else None
)

Create a Reusable Metadata Function

Instead of extracting each tag separately, you can create a reusable function:

import requests
from bs4 import BeautifulSoup


def get_meta(soup, *, name=None, property=None):
    if name:
        tag = soup.find("meta", attrs={"name": name})
    else:
        tag = soup.find("meta", attrs={"property": property})

    return tag.get("content") if tag else None


def scrape_metadata(url):
    response = requests.get(
        url,
        headers={
            "User-Agent": "Mozilla/5.0"
        },
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    return {
        "title": soup.title.string.strip()
        if soup.title and soup.title.string
        else None,

        "description": get_meta(
            soup,
            name="description"
        ),

        "keywords": get_meta(
            soup,
            name="keywords"
        ),

        "canonical": (
            soup.find("link", rel="canonical").get("href")
            if soup.find("link", rel="canonical")
            else None
        ),

        "og_title": get_meta(
            soup,
            property="og:title"
        ),

        "og_description": get_meta(
            soup,
            property="og:description"
        ),

        "og_image": get_meta(
            soup,
            property="og:image"
        ),

        "og_url": get_meta(
            soup,
            property="og:url"
        ),

        "twitter_card": get_meta(
            soup,
            name="twitter:card"
        ),

        "twitter_title": get_meta(
            soup,
            name="twitter:title"
        ),

        "twitter_description": get_meta(
            soup,
            name="twitter:description"
        ),

        "twitter_image": get_meta(
            soup,
            name="twitter:image"
        ),
    }


url = "https://example.com"

metadata = scrape_metadata(url)

for key, value in metadata.items():
    print(f"{key}: {value}")

Return Metadata as JSON

If you want to use the scraper as part of an API or another application, returning JSON is more useful.

import json

metadata = scrape_metadata("https://example.com")

print(
    json.dumps(
        metadata,
        indent=4,
        ensure_ascii=False
    )
)

Example:

{
  "title": "Example Domain",
  "description": "This domain is for use in illustrative examples.",
  "keywords": null,
  "canonical": "https://example.com/",
  "og_title": "Example Domain",
  "og_description": "Example website",
  "og_image": "https://example.com/image.jpg",
  "og_url": "https://example.com/",
  "twitter_card": "summary_large_image",
  "twitter_title": "Example Domain",
  "twitter_description": "Example website",
  "twitter_image": "https://example.com/image.jpg"
}

Scrape All Meta Tags

Sometimes you don’t know in advance which metadata tags a website uses. In that case, extract all <meta> tags:

for tag in soup.find_all("meta"):
    name = tag.get("name")
    property_name = tag.get("property")
    content = tag.get("content")

    if content:
        print(
            name or property_name,
            ":",
            content
        )

This is particularly useful when building a generic SEO metadata scraper because different websites may use different metadata conventions.

Handle HTTP Errors

Production code should handle network errors and invalid URLs.

import requests


try:
    response = requests.get(
        url,
        headers={
            "User-Agent": "Mozilla/5.0"
        },
        timeout=10
    )

    response.raise_for_status()

except requests.exceptions.Timeout:
    print("Request timed out")

except requests.exceptions.RequestException as error:
    print("Request failed:", error)

Important: JavaScript-Rendered Websites

requests downloads the HTML returned by the server. It does not execute JavaScript.

Therefore, if a website generates its metadata dynamically using JavaScript, BeautifulSoup may not see the final metadata.

For example:

Browser
   │
   ├── Download HTML
   ├── Execute JavaScript
   └── Render final page

Whereas:

Python requests
   │
   └── Download HTML only

For JavaScript-heavy websites, tools such as Playwright or Selenium can be used to load the page in a real browser before extracting the metadata.

Conclusion

For most server-rendered websites, the combination of:

requests
    +
BeautifulSoup

is sufficient for scraping SEO and social metadata.

You can extract important fields such as:

  • Page title
  • Meta description
  • Meta keywords
  • Canonical URL
  • Open Graph title
  • Open Graph description
  • Open Graph image
  • Open Graph URL
  • Twitter Card
  • Twitter title
  • Twitter description
  • Twitter image

This approach is also a good foundation for building an SEO metadata checker, URL preview generator, content crawler, or website auditing tool.