How can I scrape forums?
I need a webscraper to scrape user forums.
I'm doing this to analyze the data and quickly spot common gains, pains, jobs, quotes…
I've found a couple of them:
- https://www.zyte.com/data-extraction/
- https://www.octoparse.com/
- https://www.webscraper.io/
Questions:
- What do you use if/when you "scan" user forums? How do you do it?
- Does any of the abovementioned solutions offer any kind of visualizations?
- Is there any free alternative? Can you suggest an alternative approach if it exists?
I have tried https://apify.com in the past. Also done some scraping with puppeteer which is good.
If you know js and node you can quickly build a web scraper with puppeteer.
I tend to go for Selenium with python. it works like a charm! Also for dynamically rendered data
Free alternative - https://finddatalab.com/web-scraping and just there you can read the definition and how-to usage of scraping.
Please sign in to leave a comment.