Reporting and Analytics Developer / Data Engineer

Key Responsibilities:

• Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization
• Perform data extraction, cleaning, transformation, and data flow management. Web scraping may also be a part of the work scope in data extraction
• Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks
• Integrate and collate data silos in a manner that is both scalable and compliant
• Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to build scalable, data-driven products
• Be responsible for developing backend APIs and working on databases to support applications
• Work in an Agile environment that practices Continuous Integration and Continuous Delivery
• Work closely with fellow developers through pair programming and code review processes

Requirements:

• Proficient in general data cleaning and transformation (e.g., SQL, pandas, R, etc.) to ensure data accuracy and consistency
• Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Service (DMS), Python, AWS Lambda, ECS Container Tasks, EventBridge, AWS Glue, Spring)
• Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB, Cassandra, etc.)
• Experience in cloud technologies such as GCP, GCC (i.e., AWS, Azure, Google Cloud)
• Experience and passion for data engineering in a big data environment using cloud platforms such as GCP, GCC (i.e., AWS, Azure, Google Cloud)
• Experience with building production-grade data pipelines and ETL/ELT data integration
• Knowledge of system design, data structures, and algorithms
• Familiar with data modelling, data access, and data storage infrastructure such as Data Marts, Data Lakes, Data Virtualization, and Data Warehouses for efficient storage and retrieval
• Familiar with REST APIs and web requests/protocols in general
• Familiar with big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ)
• Familiar with W3C Document Object Model and customised web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js, etc.)
• Familiar with data governance policies, access control, and security best practices
• Comfortable with at least one scripting language (e.g., SQL, Python)
• Comfortable working in both Windows and Linux development environments
• Interest in being the bridge between engineering and analytics

Preferred Qualifications:

• Experience building data engineering pipelines that require integration with search indexes
• Experience with Airflow and RDBMS integration and implementation (e.g., MySQL)
• Experience with either Snowflake, Databricks, or an equivalent provider