Hey, it's been a while since we had any updates on our projects...

Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.

But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.

The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)

So, the part that transfers data from one database to another was decided to be open-sourced.

"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.

There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.

But, we love SQL here, right?

Let's do it like Snowflake: everything in a SQL-like language!

Snowflake has this construction called PIPE:

create pipe mypipe2 as copy into mytable(c1, c2) from (select $5, $4 from @mystage);

And then logs can be taken from a special table.

The desire was to make something similar, but more universal.

Today's no details, it's all in progress :)

I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)

Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typenoscript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.

It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.

And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?

Waiting for your ideas in the comments.

finally!

https://cron-ai.vercel.app/

I have added a new tool to my collection of non-DB SQL tools on GitHub :

https://github.com/undergrinder/gomoql
A little gomoku game, implemented in pure SQL/plpgSQL

Snowflake WTF #71:
You can name multiple CTEs with the same name, but you can only refer to the first one

For understandable reasons, I come across many articles about data catalogs, lineage, quality, and other related topics. Previously, I used to save all these links in my private chat (on Slack) and then read them (nope).

But recently, Slack warned us that they would ban us as unwanted users.

So now I have to save the links somewhere else. I thought that's a great plan, Walter, to create another channel, and share these links there. In case you're interested, here's the channel:
@allaboutdatbase

(If anyone suggests an avatar, they're awesome. I couldn't bring myself to add screenshots from Meghan Trainor's music video.)

And while I'm at it, let me tell you about a few more channels. After all, we all know that once you start collecting channels, you can't stop:

@nosingularity - all about databases (even more in the channel's chat)

@snowflake_daily - news and articles about Snowflake
@snowflakedbchat and @snowflakedbchat_en - chats about Snowflake
@snowflakejob - Snowflake job openings

@wellfirstofallitsbeautiful - beautiful things

https://twitter.com/nilansaha/status/1621421983953993728

- Railway (Free tier)
- CockroachDB (Free tier)
- mixpanel (Free tier)
- aws S3 (~ $0.60)