Data Engineering

Airflow – Fill Dagbag takes too long

Airflow – Fill Dagbag takes too long

Hung Manh Dec 14, 2022 3 min read

TLDR: It is possible to dynamically create dags with only one dag script. However, at task execution the original dag script will be parsed once again. This results in unnecessary parsing iterations of dags, which are not the parent dag…

Migrating existing OCI Kubernetes to VCN-Native Cluster with Terraform

Migrating existing OCI Kubernetes to VCN-Native Cluster with Terraform

Hung Manh Dec 2, 2022 2 min read

Your OCI Kubernetes Cluster might have a little tool tip which states “migration required”. This is because, “in earlier releases (before March 16, 2021), Container Engine for Kubernetes provisioned clusters with Kubernetes API endpoints that were not integrated into your…

Using pushdataset in PowerBI to create near real time logging dashboard

Using pushdataset in PowerBI to create near real time logging dashboard

Hung Manh Mar 1, 2022 4 min read

Recently i participated in a hackerthon, in which the goal was to create a near real time monitoring dashboard using Microsoft PowerBI. The data was already generated and persisted in SQLServer and needed to be queried efficiently. Since i am…

Reminder to update statistics

Reminder to update statistics

Hung Manh Feb 3, 2022 2 min read

The other day we had a moderate complex query which involved around 270000 rows but run for over an hour. After updating the statistics the query finished in only 4 seconds.

Kubernetes pod stuck in pending status. Nodes had no available volume

Kubernetes pod stuck in pending status. Nodes had no available volume

Hung Manh Jan 30, 2022 2 min read

A deployed pod is stuck in pending status. Looking deeper into the pod a warning appears: 0/3 nodes are available: 3 node(s) had no available volume zone.

Airflow tasks do not run at specified time as scheduled

Airflow tasks do not run at specified time as scheduled

Hung Manh Dec 9, 2021 3 min read

We observed a problem where dags did not run at the specified time at all but consistently started at a random time. Let’s dig into it.

pgadmin in Kubernetes – mount Permission denied

pgadmin in Kubernetes – mount Permission denied

Hung Manh Jul 2, 2021 2 min read

If you want to mount a PVC to your k8 cluster to persist data of your pgadmin deployment, you will likely get a mount permission denied error.

Null values and their pitfalls

Null values and their pitfalls

Hung Manh Aug 6, 2020 4 min read

Learning: Best to not permit NULL values in columns which are used for comparisons

Passing comma separated String to function / stored procedure to use with IN condition

Passing comma separated String to function / stored procedure to use with IN condition

Hung Manh Apr 28, 2020 1 min read

[..] passing comma separated variables into a function does not seem to allow the use of the variable directly in the IN condition.

hungsblog | Nguyen Hung Manh | Dresden
Scroll to Top