Objective :
Periodically, dump content of (a really) big CSV file into a Redis cache, and access the data through public API.
Processes :
1. Download CSV source file, and this process run in the background and triggered periodically.
2. Periodically, read the downloaded CSV file, parse the data, and prepare them to be saved into Redis cache.
3. Saving the data into Redis. As this process might content a huge number of "records" ,the saving process itself wrapped in the form of tasks (1 record 1 task), that will be passed into Exq queue. This task-of-saving will be excuted one by one, asynchronously.
4. Finally, once data available in Redis, the public API module will fetch them based on the API request parameter.
Source : https://github.com/dewa19/csv_dumper
Disclaimer :
1. Thanks to SpatialKey for letting me use their CSV sample file (Sample insurance portfolio) (https://support.spatialkey.com/spatialkey-sample-csv-data/)
2. The above scenario is developed as a part of learning process of understanding Elixir
