Find Jobs
Hire Freelancers

Need Node.JS / Database / Amazon AWS expert to help with Data Collection project - 03/12/2018 20:19 EST -- 2

$14-350 NZD

종료됨
게시됨 5년 이상 전

$14-350 NZD

제출할때 지불됩니다
Hi, I need an expert in Node.JS, Amazon AWS, and Database management to help with a data collection project. Project Overview: We have created a script which queries a third-party API, using a specific keyword, and returns JSON data related to that keyword. Task #1: We need that script to run once every 60 seconds. Task #2: We need a Database set up to store the JSON data. From speaking with experts, it sounds like this should be a relational database such as MySQL. Task #3 (hard part): We need to duplicate this automated script for approximately 5,000 different keywords, all running queries every 60 seconds and storing their data in the main Database. Problems: This presents a few issues, on which we need advice. Problem #1: The third-party API has a rate limit of 60 seconds per request. Presumably, this means we need to create 5,000 different scripts, all running from different IP addresses (need advice). From our research, some people have had success doing this on a smaller scale using Amazon’s Lambdas service. We need advice on how to run all of these API queries concurrently, efficiently, and at the lowest cost. Recommendations on services and solutions are welcome. Note: This is not an attempt to avoid paying for API access - it is a public API that has a rate limit. Problem #2: These queries are very quickly going to create a very large database. We need advice on the best Database service / solution to use, for maximum performance at the lowest cost. Problem #3: The goal for this data is to send it to a Front-End which will be able to present the Data in real time. We are still building that Front-End, but have a general idea of what it will look like and how the data will be presented. We need advice on the best way to Collect, Process, Analyze, Store, and Present this data in a real-time environment. From our research, there are various services and solutions for this type of problem. For example, Amazon Kinesis seems like it could be a solution? Need advice. If you are interested, please get in touch and we can chat further. Thank you for your time.
프로젝트 ID: 18289522

프로젝트 정보

5 제안서
원격근무 프로젝트
활동 중 5년 전

돈을 좀 벌 생각이십니까?

프리랜서 입찰의 이점

예산 및 기간 설정
작업 결과에 대한 급여 수급
제안의 개요를 자세히 쓰세요
무료로 프로젝트에 신청하고 입찰할 수 있습니다
5 이 프로젝트에 프리랜서들의 평균 입찰은 $390 NZD입니다.
사용자 아바타
I am working as a software developer with 10+ years of experience. Industry experience includes Big Data, Backend, DevOps (AWS and Azure) and etc. I did my MTECH in the Department of Computer Science Engineering, Indian Institute of Technology (IIT) , Guwahati, India. Some of the skills are mentioned below. Amazon Web Services: EC2, Load Balancers, RDS, Elastic Beanstalk, Athena, Glue, EMR IAM, VPC, Security Groups, Cloud Formation, Lambda, S3, CLoudFront, Redshift, Route 53, Cloud Watch, Kinesis, SNS, SQS, API Gateway and etc. Azure: VMs, SQL Databases, Storage and etc. Programming Languages: C, C++, Java, Go, Python, Bash Big Data Technologies: Hadoop, Spark, Hive, HDFS, HBase, Kafka, Flume, Memsql, Aerospike, Redis and etc. Miscellaneous: Algorithms, Data Structures, Distributed Systems, Operating Systems Databases: Oracle, MySQL, SQL Server Operating Systems: Windows, Linux, Ubuntu and etc. Code Deployments: Puppet, Docker, Airflow
$30 NZD 10일에
4.9 (23 건의 리뷰)
4.8
4.8
사용자 아바타
Hello Your problem is pretty interesting. Not sure why it requires to create 5k copies instead of single. Also why it is MySQL instead of MongoDB while you are dealing with JSON document. Thanks!
$350 NZD 3일에
4.0 (5 건의 리뷰)
3.2
3.2
사용자 아바타
ican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do thisican do this
$980 NZD 10일에
4.3 (5 건의 리뷰)
1.3
1.3

고객에 대한 정보

국기 (INDIA)
faridabad, India
5.0
35
3월 9, 2017부터 회원입니다

고객 확인

감사합니다! 무료 크레딧을 신청할 수 있는 링크를 이메일로 보내드렸습니다.
이메일을 보내는 동안 문제가 발생했습니다. 다시 시도해 주세요.
등록 사용자 전체 등록 건수(일자리)
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
미리 보기 화면을 준비 중...
위치 정보 관련 접근권이 허용되었습니다.
고객님의 로그인 세션이 만료되어, 자동으로 로그아웃 처리가 되었습니다. 다시 로그인하여 주십시오.