zaebca!

https://www.theregister.com/2023/01/16/basecamp_37signals_cloud_bill/

Hey, it's been a while since we had any updates on our projects...

Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.

But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.

The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)

So, the part that transfers data from one database to another was decided to be open-sourced.

"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.

There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.

But, we love SQL here, right?

Let's do it like Snowflake: everything in a SQL-like language!

Snowflake has this construction called PIPE:

create pipe mypipe2 as copy into mytable(c1, c2) from (select $5, $4 from @mystage);

And then logs can be taken from a special table.

The desire was to make something similar, but more universal.

Today's no details, it's all in progress :)

I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)

Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typenoscript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.

It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.

And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?

Waiting for your ideas in the comments.

finally!

https://cron-ai.vercel.app/

I have added a new tool to my collection of non-DB SQL tools on GitHub :

https://github.com/undergrinder/gomoql
A little gomoku game, implemented in pure SQL/plpgSQL

Snowflake WTF #71:
You can name multiple CTEs with the same name, but you can only refer to the first one