Data Engineering

Spark – Error with UTF8 encoding in Docker Image

Spark – Error with UTF8 encoding in Docker Image

Hung ManhMar 24, 20243 min read

In German, we encounter special characters known as Umlaute, including ä, ü, ö. If the configuration is not correctly set, encoding these symbols may result in information loss. Let’s explore a practical example where such a misconfiguration led to a…

dummy-img

Exasol – object XXX not found

Hung ManhJan 20, 20242 min read

TLDR: Identifiers in Exasol are stored in upper case internally. Selections should also be quoted. Observation: In Exasol I created a Python User Defined Function like this: CREATE OR REPLACE PYTHON3 SCALAR SCRIPT “SCHEMA”.”PARSE_XML” (“xml” VARCHAR(2000000) UTF8) EMITS (“parsed_column” VARCHAR(2000000)…

dummy-img

Python – Pass by object: Practical pitfall

Hung ManhSep 19, 20233 min read

Inside a loop I was accessing an object within a dictionary multiple times, transform and visualize it. The intention was, to have all transformation isolated from each other. What actually happened though, was that those transformations accumulated because of Python’s…

Duplicate Keys when Generating a Json from a Dictionary in Python

Duplicate Keys when Generating a Json from a Dictionary in Python

Hung ManhJan 25, 20232 min read

TLDR: A dictionary in json treats all keys as string, while a python dict distinguishes not only between the content but also its datatype (see stackoverflow). When saving a dictionary into a json and reloading the dictionary from it, you…

How To Create A Superset Guest Token With Python To Embed Dashboards

How To Create A Superset Guest Token With Python To Embed Dashboards

Hung ManhDec 30, 20224 min read

The ulterior motive is to embed a Superset Dashboard into e.g. a REACT application. To achieve this, one step includes the creation of guest tokens (service accounts). This process is (in my opinion) not sufficiently well documented, which is why…

Airflow – Fill Dagbag takes too long

Airflow – Fill Dagbag takes too long

Hung ManhDec 14, 20223 min read

TLDR: It is possible to dynamically create dags with only one dag script. However, at task execution the original dag script will be parsed once again. This results in unnecessary parsing iterations of dags, which are not the parent dag…

Migrating existing OCI Kubernetes to VCN-Native Cluster with Terraform

Migrating existing OCI Kubernetes to VCN-Native Cluster with Terraform

Hung ManhDec 2, 20222 min read

Your OCI Kubernetes Cluster might have a little tool tip which states “migration required”. This is because, “in earlier releases (before March 16, 2021), Container Engine for Kubernetes provisioned clusters with Kubernetes API endpoints that were not integrated into your…

Using pushdataset in PowerBI to create near real time logging dashboard

Using pushdataset in PowerBI to create near real time logging dashboard

Hung ManhMar 1, 20224 min read

Recently i participated in a hackerthon, in which the goal was to create a near real time monitoring dashboard using Microsoft PowerBI. The data was already generated and persisted in SQLServer and needed to be queried efficiently. Since i am…

Reminder to update statistics

Reminder to update statistics

Hung ManhFeb 3, 20222 min read

The other day we had a moderate complex query which involved around 270000 rows but run for over an hour. After updating the statistics the query finished in only 4 seconds. The query looked like so: WITH input_rows as (…

Kubernetes pod stuck in pending status. Nodes had no available volume

Kubernetes pod stuck in pending status. Nodes had no available volume

Hung ManhJan 30, 20222 min read

Observation A deployed pod is stuck in pending status. A describe pod gives the following warning: Warning FailedScheduling [..] 0/3 nodes are available: 3 node(s) had no available volume zone. What happened We already had a PVC (PVC_A), which we…

hungsblog | Nguyen Hung Manh | Dresden
Scroll to Top