Data Engineering
Airflow tasks do not run at specified time as scheduled
We observed a problem where dags did not run at the specified time at all but consistently started at a random time. Let’s dig into it. Expected Behavior: We have a job chain of three dags which are scheduled for…
How to attach block volume to VM instance in OCI
As a lot of cloud providers offer a free tier option of their products now, i wanted to run some tests on the Oracle Cloud Infrastructure (OCI) and encountered some actually trivial problems on how to increase the volume of…
pgadmin in Kubernetes – mount Permission denied
If you are using Docker, or for this matter Kubernetes, you would need to mount a host folder into your container to store your data persistently. For pgadmin, you would like to persistently save the database, which holds all metadata…
Null values and their pitfalls
Learning: Best to not permit NULL values in columns which are used for comparisons If you compare a NULL value with any other character the output will always be false which sometimes can lead to confusion since it is not…
Passing comma separated String to function / stored procedure to use with IN condition
where column = any (string_to_array(replace(delimSources,’ ‘,”),’,’)) — replace(source,old, new) => removes whitespace — string_to_array(source,separator) => puts the string separated by seperator in an array — any => equivalent to IN, BUT in contrast to IN it can take an array,…
Add leading zeroes to time
SELECT RIGHT(CONCAT(‘000000’,13000),6) When a time is saved as integer the leading zeroes will not be saved. (E.g. 1:30 am => 013000 => 13000). To add the leading zeroes we can use the above snippet. Now we can cast it to…