Urgent: Scrapy sitemap parsing gig
$25-50 USD (1시간 기준)
I have a huge list of domains that we need to parse to get all of the sitemap data out of.
I’ll provide csv of all the domains. You might need to normalize them (checking http/https protocol) and check www or not.
We need two outputs:
Summary csv with the following
Proper url to the sitemap | total pages in sitemap | list of dates for the last year and count of pages updated on those dates.
So the csv will have 367 columns
Next output I need
You can hit the sitemap for each site and dump to csv a file per domain. The csv should have the sitemap data in it.
Url / modified
I have about 160k domains that we need to process for this.
I’ll provide you a Ubuntu Aws machine to run your solution on. Thinking scrapy or similar running for a few days.
To apply for this job your proposal must include the following
1- questions
2- what framework will your solution use?
3- ballpark how much time to get the solution running?
4- how many domains per 10 sec do you think we can process?
프로젝트 ID: #22451100
프로젝트 소개
이 일자리에 대한 프리랜서 22 명의 평균 입찰가: $35 (1시간 기준)
Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl 기타
I can start work right now and I can show you perfect result in a short time. Please contact me freely. Waiting for you with your great news.
Hi, I am good in your required project; I also have a great working experience of more than 10 years. To ensure please visit my profile and check customers satisfaction level. I will complete your project within your 기타
Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON 기타
Hi. I have writen a similar app but for windows. I am ready to write your project 1- questions? Can you run it in windows? 2- what framework will your solution use? .NET 3- ballpark how much time to get the solution r 기타
Hi there! I am interested to do this project for you.'' 1- questions Ans: Please send me atleast 5 different url so i can check 2- what framework will your solution use? Ans: Scrapy will be best for this 3- ballpark h 기타
Hi. I've checked your project description and I'm interested in your job. I fully understand your requirement. I'm very skilled with: JS frameworks & libraries like Angular, React, Vue; PHP frameworks such as Laravel 기타
Hello! I have worked on a few web scraping projects in the past, some with VB and some with scrapy. To answer your questions in order: - I would use Python 3 with scrapy, and possibly modules like urllib to sanitise t 기타
Hello. I have just reviewed your job description carefully. ALL SKILLS you need have never been problem for me. Anyhow, I can solve any problem there as I have long years experience in web development. I'll be great 기타
Hi. Dear I read your job description in detail and feel I can help your project. I have full experience and skills for the python. I have done the many projects as same as your project with Flask, Django project and M 기타
Greetings. I am an expert in software architecture. I have rich experiences in machine learning, AI, image processing ,openCV and google apis and extensions. I have many experiences in programming languages such as c 기타
Hello 1- questions : please give me at least one site link and csv file for all domains. 2- what framework will your solution use? : Core PHP / DOMDocument Parser, Python scrapy framework 3- ballpark how much time 기타