Kafka, Dibizium, postgreSQL, Scrapy, Scrapy-Cluster, Kubernetes, Docker, Proxies Consultation

We are looking to get answers on some questions involving Software Architecture requiring integration of Kafka, Dibizium, postgreSQL, Scrapy, Scrapy-Cluster, Kubernetes & Docker & various web proxies or proxy services and other log-aggregation softwares with kafka.

You must be someone who has used all these tools and has done some large scale scraping using scrapy, scrapyd/scrapy-cluster, collected log data from multiple vps & fed both types (scraped & log data) data to postgresql via kafka/dibizium. And various required dashboards for monitoring & analysis.

We would require about an hour on chat and if required 15 minutes on phone. Chat will be in two stages, at different times on alternate days. You must be expert in the subject.

데이터 마이닝 PostgreSQL Python Scrapy 시스템 관리인

프로젝트 ID: #18804823

프로젝트 소개