Scaling up a Django + React Web Scraping Website
$30-250 USD
착불
I am working on a platform that runs a Django/Postgres backend with a React frontend. The platform scrapes data from multiple sources with Scrapy, cross-relates them, and gives some basic analysis of them via the front-end and HighCharts.
I am running into some issues with scaling the platform and some strange issues getting the entire system to work together. For example, running Scrapy & Twisted on a worker server has been problematic at times and I am having issues using Python multi-processing in tandem with the Django ORM (objects appear to be created randomly).
프로젝트 ID: #26870645
프로젝트 소개
이 일자리에 대한 프리랜서 6 명의 평균 입찰가: $161
Can help in scaling the scraper with scrapy. want to know more about it. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation w 기타
Why not move your script to AWS ec2 instead if you want it to run smoothly. If that doesnt work then the problem is from the script not the server
