Project Background
Digital marketing campaigns have become the primary focus for brands due to the increasing popularity of social media platforms like Instagram, Youtube, Facebook, Twitter, and others. The amount of data required for analyzing marketing performance is immense. This project aims to collect, organize, categorize, transform, and store the data streaming from various platforms into a data warehouse. By doing so, valuable business insights can be extracted from this vast ocean of data.
Challenges & Requirements
Designing the architecture and solution details for this project, issues were outlined and following points are evaluated and discussed in the team.
- Differnt formats/concepts are used in different data sources, which also make use each own way for API connection, also in vaired frequencies for requests.
- Retry logic implementation for innadvertently bad network connection.
- Most of the platfroms have set the rate limit for API request, so request throttling is needed for comply with this requirement.
- Posts need to be breakdown into keyword, mention, emoji and hashtags.
- Queue based task processing is needed for displaying the progress of data digesting tasks, and understanding the resouces usage in concurrency mode.
Docker is used for micro-services deployment and CI/CD for faster product interations. To support data scienctists, multi-layer architecture was designed.
Actions & Outcomes
To achieve the unificaton of streaming data, the adapter pattern design is used to convert each data source (platform) to the unified model which cover most of concepts. This way the objective is accomplished. Adapter module involves inheritance and concerns in Ruby.
- Retry logic was implemented by using Ruby Resque and later was refactored to using Bull.js, which provided flexible retry merchanism.
- Abstracting the time out setting into config file and adjusting the timeout value on the fly according to the returned api data, helped to throttling the requests under the limits.
- Bull.js had built-in web interface for managing tasks.
- To accommdate frequent changes and updates and build a robust codebase, testing in unity level and system are created and maintained, in order to ensure code quality and expected functionalities.
- By Regrex posts were parsed and components (mention, keyword, hashtag, emoji) were extracted.
With high testing coverage and code review, software quality is ensured. The data pipeline is able to help data scienctists to understand the business information and deriving business value from the data. Consultants are able to perform analysis, look for patterns, runn reports, provid insights and advice on an ongoing basis.
Vincent's intellectual and technical talents have been recognised consistently throughout his career. Precise, efficient and enthusiastic.
Technologies Used
Data digesting system is built in Ruby/Resque and later on refactored in Node.js/Bull.js, Redis for tasks storage. Rails is used to build web app with Postgresql as the database. Frontend includes Bootstraps, JQuery, Angular, etc. Unstructure data is kept into MongoDB for analytic purposes.
Conclusions
The data digesting micro services have been orchestrated in docker swarm in AWS. Daily data coverage over 20K brands makes sure the business insights are timely and comprehensively delivered. The unit testing and system testing have improved code design and resulted in less disruption in upgrading APIs.
