The world ain't all sunshine and rainbows. It's a very mean and nasty place.

Especially when you use select *

When a base table changes, all the downstream dependencies might break down.
Please don't deploy such things to production.

⬇️⬇️⬇️

#bugs

AWS denounces its own error logs

Your post may include a non-inclusive word (master)

😭😂

Hey, long time no see 🙂

We have just started our engagement with dbtLabs at Wheely.

Guys will try to conduct audit and help us improve our dbt deployment even further.

They have already access to:
– dbtCloud (jobs)
– Github repo (code)
– Redshift (database)
– Looker (bi + monitoring)
– Slack (communicating)

And yesterday was the Kick-off call. Overall great impressions.

What is going to happen:
– Audit: Deployment + Performance
– Audit: Project Structure
– Features/Fix's

I will keep you posted.

Airbyte Clickhouse destination

Airbyte deployed Clickhouse destination which I already use to gather data from multiple sources.

By default it replicates all the data as JSON blobs (all the attributes inside one String field)

To get it flattened you either have to do it yourself or use Airbyte normalization.

1. Flattening manually with JSON functions

JSONExtract(_airbyte_data, 'id', 'UInt64') as id

➕ Could be tricky and exhausting if you have a lot of attributes.

➖ Works extremely fast

2. Airbyte normalization (= dbt underneath 😉)

➕ It will flatten all your data automatically

Technically it is auto-generated dbt project

➖ Still a little bit buggy and looks like a work in progress.

I almost managed to get it done, but I use Yandex’ Managed Clickhouse which demands you use SSL / Secure connection.

Unfortunately, Airbyte’s dbt profiles.yml is hard-configured to secure: False at the moment.

I might create a PR to fix this when I have some time.

#airbyte #clickhouse #dbt

I will try to overcome normalization another day 😄

Leave a comment / reaction if you are interested

Datalens from Yandex is quite powerful BI tool.

Especially when you use it on top of Clickhouse which makes analytics interactive with sub-second latency.

Amongst outstanding features I've already tried:

— Advanced functions to built almost anything one can imagine: timeseries, arrays, geo, window functions
— Nice and customizable charts integrating with dashboard
— Sharing with team / anyone on the internet

The more I use it, the more I love it.

— Useful docs with examples and how-to
— Really friendly community here in Telegram (important!)
— It is free of charge!

Take a look at how I managed to build Year-over-Year analysis with LAG function and draw different kinds of viz!

LAG([Выручка (₽)], 52 ORDER BY [Неделя] BEFORE FILTER BY [Неделя])

#datalens #clickhouse

Привет! Новая публикация на Хабр ⬇️⬇️⬇️
Накиньте плюсов, если материал нравится, а я уже готовлю вторую часть.