Data Engineering
Airflow – Fill Dagbag takes too long
TLDR: It is possible to dynamically create dags with only one dag script. However, at task execution the original dag script will be parsed once again. This results in unnecessary parsing iterations of dags, which are not the parent dag…
Migrating existing OCI Kubernetes to VCN-Native Cluster with Terraform
Your OCI Kubernetes Cluster might have a little tool tip which states “migration required”. This is because, “in earlier releases (before March 16, 2021), Container Engine for Kubernetes provisioned clusters with Kubernetes API endpoints that were not integrated into your…
Using pushdataset in PowerBI to create near real time logging dashboard
Recently i participated in a hackerthon, in which the goal was to create a near real time monitoring dashboard using Microsoft PowerBI. The data was already generated and persisted in SQLServer and needed to be queried efficiently. Since i am…
Reminder to update statistics
The other day we had a moderate complex query which involved around 270000 rows but run for over an hour. After updating the statistics the query finished in only 4 seconds.
Kubernetes pod stuck in pending status. Nodes had no available volume
A deployed pod is stuck in pending status. Looking deeper into the pod a warning appears: 0/3 nodes are available: 3 node(s) had no available volume zone.
Airflow tasks do not run at specified time as scheduled
We observed a problem where dags did not run at the specified time at all but consistently started at a random time. Let’s dig into it.
pgadmin in Kubernetes – mount Permission denied
If you want to mount a PVC to your k8 cluster to persist data of your pgadmin deployment, you will likely get a mount permission denied error.
Null values and their pitfalls
Learning: Best to not permit NULL values in columns which are used for comparisons
Passing comma separated String to function / stored procedure to use with IN condition
[..] passing comma separated variables into a function does not seem to allow the use of the variable directly in the IN condition.