Join us as we follow Tee through a day in the life of a data engineer at Sertis.
Some of you may not be familiar with the role of a data engineer, unlike positions such as data analysts and data scientists. However, a data engineer is the backbone of successful data-driven projects.
Today, we introduce Tee, a talented data engineer at Sertis. Tee is a valued member of our compact data engineering team, responsible for constructing data pipelines that are essential to the success of numerous large-scale projects.
Explore Tee's daily routine as a Data Engineer at Sertis, his tasks, the tools he utilizes, and what sparks his passion for the data engineer role.
What are the daily tasks of a data engineer?
"Firstly, let's clarify what data engineering is all about. 'Data' refers to information, and 'engineering' means to design and build a system. When combined, data engineering refers to the practice of designing and building data systems.
Typically, data engineering tasks are known to revolve around ETL, which is abbreviated from:
‘E’ for extracting data from various sources.
‘T’ for transforming data into an organized and readily available format.
‘L’ for loading data into storage like a database, making it available for data analysts, data scientists, or other data users to utilize.
So, the daily tasks of a data engineer are primarily focused on the ETL process. A data engineering team gets to collaborate with various teams, particularly the Consulting & Delivery team, which is responsible for project oversight and task allocation. Additionally, we work closely with data analysts who provide data modeling requirements, enabling us to design data pipelines tailored to these specifications.”
What does a day in the life of a data engineer look like?
9:00 AM - 10:00 AM - Breakfast and Coffee
I like to think of this time as setting the stage for the rest of the day. I start my morning with breakfast and a cup of coffee to fuel up, much like filling up a gas tank before embarking on a journey.
10:00 AM - 11:00 AM - Project-based Daily Stand-up Meeting
This hour is dedicated to project-based daily stand-up meetings, where we discuss progress and sync up with the project team. Each meeting lasts around 15-30 minutes. If I'm assigned to three projects, it can take up to 30 minutes to 1 hour. These meetings are an essential part of my day.
1:00 PM - 7:00 PM - Project-based Tasks, Team Meetings, and more
Following lunch, I work on project-based tasks, which may include internal meetings, client meetings, and other development-related activities. Sometimes, there's a data engineering team stand-up meeting to sync up with the team. Also, I may have one-on-one meetings with seniors or managers to update them on my progress and discuss my well-being, career path, additional skills I want to learn, and their expectations of me.
What are the typical tools a data engineer uses?
"I'd like to categorize data engineering tools into two main groups. Firstly:
Programming Languages: My main tools are Python and SQL. Python is a general-purpose language widely used for data engineering tasks as it's packed with handy tools and libraries for data analytics, such as Pandas and NumPy. SQL, on the other hand, is a query language utilized for interacting with data, posing questions, and getting desired answers.
Another group is data management tools, which can be subsequently divided as follows:
Database Management System (DBMS): When it comes to SQL-based DBMS, we use PostgreSQL, and for NoSQL, we opt for MongoDB. At Sertis, our primary tools are cloud-based. For instance, Google Cloud SQL serves as a smaller-scale DBMS, and Google Cloud BigQuery functions as a data warehousing tool for storing large volumes of data, querying data for analysis, and executing machine learning models.
Orchestration: Orchestration is the tool for managing the entire workflow. We mainly rely on Apache Airflow, an open-source workflow management platform that enables us to define steps in our data pipeline. As for cloud-based tools, we use Cloud Composer.
Data Processing: Tasks such as data cleaning and transformation are handled using Apache Spark, an open-source cluster computing framework designed for managing Big Data by distributing workloads across different nodes in a cluster. Also, we primarily use Dataproc for cloud-based solutions.”
What do you like the most about being a data engineer?
“What I like about the role is that it often goes unnoticed. When people ask me about my occupation, I describe myself as a person who makes data available. While my job may not stand out as that of a data analyst or a data scientist, who extracts insights from data and presents results to users, a data engineer is like the person behind the scenes, serving as the backbone that enables these insights to become a reality.
A data engineer acts as the glue that binds various teams and components within the system to allow for effective collaboration, which means I often get to collaborate with various teams. With my small team of 4-5 people, we can build and maintain a system that powers large-scale businesses.
Being a data engineer allows me to improve my business acumen. I have to understand what clients want from the data. Therefore, when designing and determining each step of my work, I do so with the business requirements in mind."
How do you see yourself in the next 2 years?
"I see myself continuing as a data engineer, enhancing my skills, and acquiring more experience to boost my confidence and enable me to lead projects independently. I wish to participate in large-scale projects that have a real-world impact, such as environmental initiatives or projects that provide valuable insights to large companies.
Also, I hope that in the next 2 years, I will still enjoy being a data engineer, be determined to learn from others, gather sufficient skills and experiences to mentor junior colleagues and share insights with my coworkers. That's my current aspiration."
For more information about Sertis, including job opportunities, insights into our culture, and a glimpse into life at Sertis, visit our website: https://www.careers.sertiscorp.com/