3. CSV Dumper + Fast API Access

Objective :
Periodically, dump content of (a really) big CSV file into a Redis cache, and access the data through public API.

Processes :
1. Download CSV source file, and this process run in the background and triggered periodically.
2. Periodically, read the downloaded CSV file, parse the data, and prepare them to be saved into Redis cache.
3. Saving the data into Redis.  As this process might content a huge number of "records" ,the saving process itself wrapped in the form of tasks (1 record 1 task), that will be passed into Exq queue. This task-of-saving will be excuted one by one, asynchronously.
4. Finally, once data available in Redis, the public API module will fetch them based on the API request parameter.

Source : https://github.com/dewa19/csv_dumper

Disclaimer :
1. Thanks to SpatialKey for letting me use their CSV sample file (Sample insurance portfolio) (https://support.spatialkey.com/spatialkey-sample-csv-data/)
2. The above scenario is developed as a part of learning process of understanding Elixir 

2. Lookup_Email : An Email Validator

Objective :

The objective of this project is to validate an email address from the side of SMTP server, without really sending a message to given email. If -by sending a series of SMTP commands- an email is acknowledged by its SMTP server, then it is an "valid" email.

Processes :
1. Fetch domain name of the email
2. Verify the domain
3. If its verified, send some SMTP commands
4. Based on its responses, we take an "educated guess" about the status of a given email

Source : https://github.com/dewa19/lookup_email

1. API Service with Redis

Objective :
Create a fully functional scenario of an API architecture using Redis as a cache mechanism. End user will have a benefit of fast response from in-memory database like Redis.

Internal (in house process) :
1. Read from 3rd party Provider (BART API, in this case)
2. Parse, save to database and Redis cache

Public API (exposed to end user) :
1. User API request
2. Read from Redis cache engine
3. If it can't find it in Redis, read from database (reply back) and save a copy into Redis for the next API requests.

Disclaimer :
1. 3rd party Data Provider, referring data provided by https://www.bart.gov/schedules/developers/api
2. The above scenario is developed as a process of learning Elixir 

Disclaimer

This is my personal notes on Erlang & Elixir that serve as Backend Services. This  note is the way I remembering everything I've learned along the way.

Unless stated otherwise, all projects presented here are my personal project and for educational purpose only. However, the scenario in each of them might be applicable in real world condition, of course with some adjustments.

It is possible, in some occasion, I'll write about real word projects("anonymously") with detail step-by-step explanations.