Web site scraping / data extraction agent, API and database for commercial aggregation service

We are launching a new commercial data aggregation service that is identical to Yodlee but designed to handle a different data set. Instead of extracting and serving consumer card and bank transaction data, we will extract and serve merchant account transaction data (the data related to the merchant side of a card transaction, which includes elements such as the fees paid to process the transaction, batch data etc.). Merchant transaction data is much more rich in terms of the number of data elements and how various data is related to each other. We are looking for a highly qualified freelancer to develop the system starting with a single site. From there we will start adding more sites to be harvested once we are comfortable with the system. The system we need developed will have three components as follows. In terms of the technology stack, we plan and prefer to host with Microsoft Azure but are also willing to consider hosting at AWS. We want a turnkey solution such that it can be deployed in our hosting and working without requiring us to write and code. Data Extraction Agent This would be the agent, which would extract data from merchant account websites just as Yodlee extracts data from consumer bank accounts. The agents must be scalable and able to harvest data from a large number of different merchant accounts at one time. While we will start with a single agent for a single site, the plan is to development dozens of agents for dozens of sites so the system must be flexible. All merchant accounts will require credentials (username and password) for each individual user, which will be stored in our database. Further, the agent must be able to handle challenge questions, which can vary by user and by site, used to authenticate a user. The agent must be able to accommodate complex sites that support AJAX and other technologies. The agent will gather the data as prescribed by us and populate a database. The agent must be able to do timed batch jobs as well as real time jobs. There does not need to be a UI for the agent as we will serve all data through the API below. API In order for our customers (financial software companies and banks) to be able to access the data we require an API to be developed. The API must include the ability to set up authorized users with authentication to validate who is making the request. When called, the API will trigger the appropriate agent using the appropriate individual user credentials to begin harvesting a site. Once harvested, the data will be passed back to the requestor and also added to the database as listed below. We need a turn key API with server implementation. Database We need a database developed to store the data extracted by the agent both in batch and real-time jobs. We would like to us Microsoft SQL but are open to discussing another database. Enclosed is a screen shot of one of the pages with some data removed for privacy. There are ten screens total most with much less fields that we would like to extract. We are happy to provide a login if you require more information to size the project. This site does support exporting the data which can be used as a method for extraction versus screens carping but the system should support both screen scraping and file extraction.