Forwarded from Технологический Болт Генона
demo.gif
24.9 MB
Запилили балалайку, которая позволяет исправлять и объяснять ошибки с помощью GPT-3.
1. Вставляем код
2. Вставляем текст ошибки
. . .
5. PROFIT11!!!
An AI-powered debugger
https://github.com/shobrook/adrenaline/
Попробовать тут
https://useadrenaline.com/playground
Обсуждение на Reddit
https://www.reddit.com/r/MachineLearning/comments/106q6m9/p_i_built_adrenaline_a_debugger_that_fixes_errors/
1. Вставляем код
2. Вставляем текст ошибки
. . .
5. PROFIT11!!!
An AI-powered debugger
https://github.com/shobrook/adrenaline/
Попробовать тут
https://useadrenaline.com/playground
Обсуждение на Reddit
https://www.reddit.com/r/MachineLearning/comments/106q6m9/p_i_built_adrenaline_a_debugger_that_fixes_errors/
There will be no singularity
Сейчас все пилят тиктоки свои БД. Помните про firebolt , да? Подписчик поделился ссылкой на проект своих друзей - unum.am Говорят, уделывает FoundationDB и все остальные noSQL. Правда, cloud only, без опенсорсов. Еще наткнулся на новую графовую базу - memgraph.com…
Remember these guys?
The guys from Unum have spent 2022 polishing their Key-Value Store, integrating SPDK, and starting an Open-Source ecosystem around it! The project UKV brings a shared compatibility layer between LevelDB, RocksDB, and their persistent and in-memory Key-Value Stores, some of which are also public.
- UKV can wrap any one of those Stores into a Standalone server with an Apache Arrow Flight RPC protocol, making it available across the network.
- Similar to Redis, with RedisGraph, RedisJSON, and RediSearch, UKV add supports for Graphs, Documents, and Vectors/Features. Unlike Redis, you still get ACID transactions and Strictly Serializable Consistency Guarantees. Document collections, for example, support JSON, BSON, MessagePack, and contents, as well as JSON-Pointer, JSON-Patch, and JSON-MergePatch RFC standards. Also, with the Embedded variant, you don't have to pay for Networking.
- UKV also brings structured bindings for high-level bindings. Python Graph bindings, for example, look like NetworkX. Python Tabular interface aims to become compatible with Pandas over time. In both cases, the query will operate on the entire dataset stored on disks, which can reach 100 TB+.
In 2023 guys are promising to work on replication and sharding, so stay tuned!
Designing the fastest ACID Key-Value Store / Ashot Vardanian (Unum cloud): https://www.youtube.com/watch?v=ybWeUf_hC7o
PS: The author is among us, you can ask questions in the comments to this post
The guys from Unum have spent 2022 polishing their Key-Value Store, integrating SPDK, and starting an Open-Source ecosystem around it! The project UKV brings a shared compatibility layer between LevelDB, RocksDB, and their persistent and in-memory Key-Value Stores, some of which are also public.
- UKV can wrap any one of those Stores into a Standalone server with an Apache Arrow Flight RPC protocol, making it available across the network.
- Similar to Redis, with RedisGraph, RedisJSON, and RediSearch, UKV add supports for Graphs, Documents, and Vectors/Features. Unlike Redis, you still get ACID transactions and Strictly Serializable Consistency Guarantees. Document collections, for example, support JSON, BSON, MessagePack, and contents, as well as JSON-Pointer, JSON-Patch, and JSON-MergePatch RFC standards. Also, with the Embedded variant, you don't have to pay for Networking.
- UKV also brings structured bindings for high-level bindings. Python Graph bindings, for example, look like NetworkX. Python Tabular interface aims to become compatible with Pandas over time. In both cases, the query will operate on the entire dataset stored on disks, which can reach 100 TB+.
In 2023 guys are promising to work on replication and sharding, so stay tuned!
Designing the fastest ACID Key-Value Store / Ashot Vardanian (Unum cloud): https://www.youtube.com/watch?v=ybWeUf_hC7o
PS: The author is among us, you can ask questions in the comments to this post
everything is better with bluetooth rust. even a python…
https://twitter.com/aberges4/status/1614226340349108224
https://twitter.com/aberges4/status/1614226340349108224
Twitter
Processing 800k lines CSV file in #pandas vs #duckdb vs #polars, involving a fairly long regex:
pandas: 9.32s
DuckDB: 3.05s
polars: 120ms
wow.
pandas: 9.32s
DuckDB: 3.05s
polars: 120ms
wow.
There will be no singularity
В эфире наша постоянная рубрика «шо там у рубистов» https://news.1rj.ru/str/oleg_log/2697 So let’s talk about what we spend on our programming language and web framework. It’s about 15% Дальше идет сухой расчет, что это всего какие-то $450k в год, «у нас в долине…
The Register
Basecamp details 'obscene' $3.2 million bill that caused it to quit the cloud
Reckons Dell kit could have stretched that dollar further
Hey, it's been a while since we had any updates on our projects...
Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.
But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.
The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)
So, the part that transfers data from one database to another was decided to be open-sourced.
"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.
There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.
But, we love SQL here, right?
Let's do it like Snowflake: everything in a SQL-like language!
Snowflake has this construction called PIPE:
The desire was to make something similar, but more universal.
Today's no details, it's all in progress :)
I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)
Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typenoscript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.
It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.
And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?
Waiting for your ideas in the comments.
Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.
But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.
The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)
So, the part that transfers data from one database to another was decided to be open-sourced.
"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.
There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.
But, we love SQL here, right?
Let's do it like Snowflake: everything in a SQL-like language!
Snowflake has this construction called PIPE:
create pipe mypipe2 as copy into mytable(c1, c2) from (select $5, $4 from @mystage);And then logs can be taken from a special table.
The desire was to make something similar, but more universal.
Today's no details, it's all in progress :)
I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)
Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typenoscript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.
It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.
And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?
Waiting for your ideas in the comments.