Data Engineering
Using pushdataset in PowerBI to create near real time logging dashboard
Recently i participated in a hackerthon, in which the goal was to create a near real time monitoring dashboard using Microsoft PowerBI. The data was already generated and persisted in SQLServer and needed to be queried efficiently. Since i am…
Reminder to update statistics
The other day we had a moderate complex query which involved around 270000 rows but run for over an hour. After updating the statistics the query finished in only 4 seconds. The query looked like so: WITH input_rows as (…
Kubernetes pod stuck in pending status. Nodes had no available volume
Observation A deployed pod is stuck in pending status. A describe pod gives the following warning: Warning FailedScheduling [..] 0/3 nodes are available: 3 node(s) had no available volume zone. What happened We already had a PVC (PVC_A), which we…
Airflow tasks do not run at specified time as scheduled
We observed a problem where dags did not run at the specified time at all but consistently started at a random time. Let’s dig into it. Expected Behavior: We have a job chain of three dags which are scheduled for…
How to attach block volume to VM instance in OCI
As a lot of cloud providers offer a free tier option of their products now, i wanted to run some tests on the Oracle Cloud Infrastructure (OCI) and encountered some actually trivial problems on how to increase the volume of…
pgadmin in Kubernetes – mount Permission denied
If you are using Docker, or for this matter Kubernetes, you would need to mount a host folder into your container to store your data persistently. For pgadmin, you would like to persistently save the database, which holds all metadata…
Null values and their pitfalls
Learning: Best to not permit NULL values in columns which are used for comparisons If you compare a NULL value with any other character the output will always be false which sometimes can lead to confusion since it is not…
Passing comma separated String to function / stored procedure to use with IN condition
where column = any (string_to_array(replace(delimSources,’ ‘,”),’,’)) — replace(source,old, new) => removes whitespace — string_to_array(source,separator) => puts the string separated by seperator in an array — any => equivalent to IN, BUT in contrast to IN it can take an array,…
Add leading zeroes to time
SELECT RIGHT(CONCAT(‘000000’,13000),6) When a time is saved as integer the leading zeroes will not be saved. (E.g. 1:30 am => 013000 => 13000). To add the leading zeroes we can use the above snippet. Now we can cast it to…
