E-commerce traffic management system

About the client

European e-commerce project with extensive commercial experience and traditions of doing business, successful partnerships, and with companies to position itself as one of the leading suppliers of smart web traffic management solutions.

Through collaboration with over 52 European businesses, among them enterprise firms and government agencies, this firm excels in traffic arbitrage which helps partners attract possible customers and increase their annual capital turnover by millions of euros. Its powerful server infrastructure can handle hundreds of customer requests at a time and store millions of product items in its database resulting in efficient and reliable service delivery.

By having these abilities, the business expands its client partnership networks thereby capturing a significant market share of the web traffic market.

Challenge

• Increased demand and broader alliances: during the past financial year, the customer’s services recorded a higher demand which fueled a rapid growth of partnerships. Consequently, this built up an increasing product volume that was processed by the system, thereby dramatically overloading AWS servers and MongoDB databases.
• System Overload and Failure: More loads brought about processing delays with increased information systems crashing frequency. Sometimes the vast database costing several millions could not handle incoming data causing shutdowns of AWS servers.
• Data Management Challenges: The client had problems with outmoded or obsolete data stored on its computers. Current issues necessitated archiving and other storage arrangements for statistics analysis with regard to assuring consistent operation of a system without overload.
• Obtaining damaged/incomplete data from partners: Without a robust check in place for partner data coming in, information comprehensiveness and integrity became greatly compromised; hence making it impossible to process the correct data and integrate into MongoDB or AWS cloud.

Solution

• Our engineers were able to optimize the operation of the ABC servers and the MongoDB database, switched to a new version of PHP that enabled faster work with the server part of the project;
• It was possible to find a bottleneck in the mongodb database that reduced the throughput of data processing in the database, separated the servers from the cron job, which provided more stable system operation and reduced the time of data processing and retrieval, and reduced delays to 0, and server transitions to offline mode due to overload fell sharply;
• Engineers were able to break down legacy data by year for better handling, created a script for archiving, and performed a data packaging procedure to improve database performance;
• A system for monitoring received and sent data was developed and implemented to improve analysis, data quality collection, and filtering of damaged/incomplete files that are processed by the system and sent to partners.

Team

Initially, the team included 1 developer and PM and over time grew to 4 people:

• Full stack engineer/ Team Leader;
• Senior Full Stack Engineer;
• DevOps Engineer;
• Project Manager.

Project stack

Key Outcomes

• Engineers updated the servers’ environment, infrastructure, and architecture of the database system, which let the system handle higher volumes of data
• The traffic grew x3 in 4 months, which allowed the client to process over 5 million transactions daily