Feb 2019 - Present
Migrated reporting sub-system to Flink, simplifying architecture and reducing data latency from daily to near real-time, enabling multi-dimensional reporting. (Flink)
Designed and deployed Airflow infrastructure, optimizing ETL workflow management for improved efficiency. (Airflow)
Refactored DSP platform data pipeline, doubling system capacity and improving stability. (Kafka, Kafka Connect, OLAP, OTAP, AWS Redshift, AWS S3)
Developed Audience Profile Service and ETLs, enhancing DSP audience targeting with scalable data processing. (Python, Spark, Scala, HyperLogLog, Aerospike)
Built and optimized a payment rule engine, enabling near real-time profit sharing calculations. (Python, Scala, Drools, Rete Algorithm)