R nGPT: Normalized Transformer with Representation Learning on the Hypersphere

Paper: https://arxiv.org/pdf/2410.01131

Abstract:

>We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Highlights:

>Our key contributions are as follows:

Optimization of network parameters on the hypersphere We propose to normalize all vectors forming the embedding dimensions of network matrices to lie on a unit norm hypersphere. This allows us to view matrix-vector multiplications as dot products representing cosine similarities bounded in [-1,1\]. The normalization renders weight decay unnecessary.

Normalized Transformer as a variable-metric optimizer on the hypersphere The normalized Transformer itself performs a multi-step optimization (two steps per layer) on a hypersphere, where each step of the attention

/r/MachineLearning
https://redd.it/1g0lnij

PSA: If you're starting a new project, try astral/uv!

It's really amazing, complex dependencies are resolved in mere miliseconds, it manages interpreters for you and it handles dev-dependencies and tools as good if not better than poetry. You are missing out on a lot of convenience if you don't try it. check it out here.

Not affiliated or involved in any way btw, just been using it for a few months and am still blown out of the water by how amazing uv and ruff are.

/r/Python
https://redd.it/1g0imjf

Introducing Eventum ASGI, a Python framework simplifying the creation of WebSocket-based apps

# Introduction:

I'm excited to present my first Python framework. I would appreciate any feedback you could give me, it's my first project and it's still in active development.

# What My Project Does:

The project is based on ASGI protocol, the key idea is to simplify the usage of WebSockets which isn't a strong side of most popular frameworks. The framework introduces some new approaches to handling WebSockets, most of the time you'll work with a WSConnection class which is one of the keystones of the framework.

Another significant difference from the common approach is the connection lifecycle in the app.

1. You create a handshake_route, which is only responsible for handling the initial request. It expects to get a handshake request to switch protocols.
2. You create an event. To make it easier to understand you can also consider it to be a route, just for messages sent via an established connection. It expects a JSON which must contain an "event" field in it.

To explain how everything works behind the scenes:

1. A client sends a handshake to switch protocols and for a server to either accept or reject a connection.
2. Once accepted, the connection gets into a loop where it's constantly checking

/r/Python
https://redd.it/1g0px5z

Generating nice iPython notebooks diffs with Git pre-commit hooks

https://preview.redd.it/u4e94ccihztd1.png?width=897&format=png&auto=webp&s=28a6d23da591912c6aa712556731798ddbfa9c7c

I like to use iPython notebooks to store experimental code and debugging results, but it's a pain to use version control to look at them.

So I wrote some pre-commit hooks that makes it easy to diff iPython notebooks in Git. It auto-generates a copy of the file with just the Python code, so that you can just inspect code changes.

I wrote a bit more about why here, along with instructions on how to use them: https://blog.moonglow.ai/diffing-ipython-notebook-code-in-git/

And the git repo for the hooks (MIT-licensed) is here: https://github.com/moonglow-ai/pre-commit-hooks

/r/IPython
https://redd.it/1g0rjpo

Deploying (Multiple) Django Apps to a Single Server with Kamal 2
https://www.coryzue.com/writing/kamal-django/

/r/django
https://redd.it/1g0i0nl

In a API Rest World, what do you choose? Blueprints or Flask-Views? Why?



/r/flask
https://redd.it/1g0sl6t

Returning dynamic form elements when invalid

I have a potentially 3 level form that can have elements added to it (either forms or formsets).

If any form (or formset) is invalid during the post, I'd like to be able to return the whole form (including dynamic content) to the user for correction before resubmission.

The second level is pretty straight forward as that is just a formset that can be rendered beneath the base form.

However, the third level is where I'm having difficulty as that is a formset belonging to a form of a formset.

The below code snippet shows the issue:

monthly_formset = MonthlyActivityDaysFormset(request.POST, prefix='m')
if monthly_formset.is_valid():
for monthly_form in monthly_formset:
##DO STUFF
if monthly_form.prefix+'-diff_times_per_month_monthly' is not None:
    diff_times_formset = DifferentTimesFormset(request.POST, prefix=monthly_form.prefix+'-dt')
      if diff_times_formset.is_valid():
                                for diff_times in diff_times_formset:
                                 

/r/djangolearning
https://redd.it/1g10hqo

Friday Daily Thread: r/Python Meta and Free-Talk Fridays

# Weekly Thread: Meta Discussions and Free Talk Friday 🎙️

Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!

## How it Works:

1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.

## Guidelines:

All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.

## Example Topics:

1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.

Let's keep the conversation going. Happy discussing! 🌟

/r/Python
https://redd.it/1g0ww31

How to connect MySQL database to flask app

Very begginer in flask and MySQL in general and I’ve been having trouble in connecting my database to the Flask app. It’s a very simple login page where the user id and authentication key per user is already inside the database, so the program has to confirm whether or not the inputted user id and authentication key are inaide the database to allow the user to access their dashboars. I’ve mostly been relying on youtube but I can’t seem to find the right one I’m looking for.

If anyone could suggest any references or suggestions that would be very much appreciated.

/r/flask
https://redd.it/1g14umt

Pyinstrument v5.0 - flamegraphs for Python!

Hi reddit! I've been hard at work on a new pyinstrument feature that I'm really excited to show off. It's a completely new HTML renderer that lets you see visually exactly what happened as the program was running.

What it does First, some context: Pyinstrument is a statistical profiler for Python. That means you can activate it when you're running your code, and pyinstrument will record what happens periodically, and at the end, give you a report that tells you where the time was spent.

Target Audience Anyone wondering if their Python program could be faster! Not only is it useful from a performance perspective, it's also a nice way to understand what's going on when a program runs.

Comparison If you've used profilers like cProfile before, pyinstrument aims to be a more user-friendly, intuitive alternative to that. It's also a statistical profiler, it only samples your program periodically, so it shouldn't slow the program down too much.

So, what's new? Up until now, the output has been some form of call stack. That's great to identify the parts of code that are taking the most time. But it can leave some information missing - what's the pattern of the code execution? What order

/r/Python
https://redd.it/1g1az6i

Tkinter based package for sending GUI alerts / notifications, named tk-alert

Hi everyone, I have been thinking to post here for some time and decided to do so. I was hesitant as this is my first time working on a python package and the project is far from being finished.

Long story short, I have been working on a personal app using Tkinter and I needed a way to send error notifications to users, could not find something really easy to install and use so I started working on creating my own self-contained package.


1. What my project does.

Sends GUI notifications for users meant for information, warnings or errors, using Tkinter.

My design philosophy was that the package should be simple and ready to use out-of-the-box, but should have more complex design time features for people that want a specific look on their app (this part is work in progress)

So far I did not have time to continue work on this due to multiple reasons, but as the cold season approaches I am looking forward to get on with some tasks from my to-do list.

2. Target audience.

Tkinter devs, not ready for production yet.

3. Comparison.

What I want this package to be set apart by is the ease of set-up and use +

/r/Python
https://redd.it/1g17jeq

Thoughts on hosting

Hello!

I've got experience with hosting wagtail/Django on heroku, I liked how easy it is to set things up and add postgres db for example.

Do you have any recommendations based on ease of use and cost? :) thanks

/r/django
https://redd.it/1g186wy

Random context generator - RaCoGen (provisional name)

What my project does:

RaCoGen is a simple program that generates a random context (a situation in which then two characters are put) by making use of 3 databases (nouns, adjectives and actions).

1. First, it selects a random noun and adjective, and it generates a setting with that, like "big forest" or "sandy gym".
2. Then, it selects and action, like "talking", "drawing"...
3. Finally, it generates the context using the setting and action, giving a result like "In a sandy gym, where char1 and char2 are drawing."

After all of this is ready, the program prints the result like this:

Random noun selected: beach

Random adjective selected: cultural

Random setting created: cultural beach

Random action selected: sleeping

Random context created: In a cultural beach, where char1 and char2 are sleeping.

Target audience:

This project doesn't have a target audience in mind because it's an experiment. I'm just seeing what I can or can't do. You can consider it a toy, because it's more for entertainment than anything eslse.

But that's just for now. I will, probably, expand this so it gives the users more options, has more variety, etc.

For now, it's made to test while I learn, but maybe in the future it could turn to an app

/r/Python
https://redd.it/1g1e7as

Automatic Flowcharts

Are there any tools or libraries that make automatic flowcharts or something similar? Like the call stack when debugging but more like a diagram of all the calls that are made since an if name == '__main__' is executed. It would be useful to see more or less what a program does even if it is not completely accurate.

/r/Python
https://redd.it/1g190dl

Web Developer Job Opportunity

Hi, we have listed a new job on our platform so if you are looking for a Web Developer job please check the below link



Role - Web Developer / Remote / East Coast 🧑‍💻 (Remote, Full-Time) 🚀



Job Link - https://devloprr.com/jobs#314

https://preview.redd.it/jafvhphdyvtd1.png?width=989&format=png&auto=webp&s=f72d1448ce1077926d9e037b96552ee7a3695298




/r/flask
https://redd.it/1g0dxpg

Do I need Nginx if I have a Network Load Balancer in front of Gunicorn / Flask?

I am new to Gunicorn and have seen that it is highly recommended to have it behind some sort of reverse proxy, namely Nginx.

The backend of the application is just a REST API and does not serve any static files. It is running in a private subnet, so the NLB is being used to forward traffic to it. I am not sure how I would do this with Nginx...or even if it is necessary in this case as I already have a load balancer.

Any input or thoughts on this would be appreciated.

/r/flask
https://redd.it/1g1m7pg

Saturday Daily Thread: Resource Request and Sharing! Daily Thread

# Weekly Thread: Resource Request and Sharing 📚

Stumbled upon a useful Python resource? Or are you looking for a guide on a specific topic? Welcome to the Resource Request and Sharing thread!

## How it Works:

1. Request: Can't find a resource on a particular topic? Ask here!
2. Share: Found something useful? Share it with the community.
3. Review: Give or get opinions on Python resources you've used.

## Guidelines:

Please include the type of resource (e.g., book, video, article) and the topic.
Always be respectful when reviewing someone else's shared resource.

## Example Shares:

1. Book: "Fluent Python" \- Great for understanding Pythonic idioms.
2. Video: Python Data Structures \- Excellent overview of Python's built-in data structures.
3. Article: Understanding Python Decorators \- A deep dive into decorators.

## Example Requests:

1. Looking for: Video tutorials on web scraping with Python.
2. Need: Book recommendations for Python machine learning.

Share the knowledge, enrich the community. Happy learning! 🌟

/r/Python
https://redd.it/1g1npyb

D Why does it seem like Google's TPU isn't a threat to nVidia's GPU?

Even though Google is using their TPU for a lot of their internal AI efforts, it seems like it hasn't propelled their revenue nearly as much as nVidia's GPUs have. Why is that? Why hasn't having their own AI-designed processor helped them as much as nVidia and why does it seem like all the other AI-focused companies still only want to run their software on nVidia chips...even if they're using Google data centers?

/r/MachineLearning
https://redd.it/1g1okem

N Kaido Orav and Byron Knoll's fx2-cmix Wins 7950€ Hutter Prize Award!

Kaido Orav and Byron Knoll have improved 1.59% on the Hutter Prize for Lossless Compression of Human Knowledge with their "fx2-cmix" entry.Because the Hutter Prize restricts contestants to a single general purpose processor and it uses the most *general* loss function, the required algorithmic advances are generally applicable regardless of the industry's "Hardware Lottery" or loss function compromises.  In this respect it provides a unique and low risk incentive for scientific advancement in machine learning.

Some of fx2-cmix's algorithmic advances over the prior Hutter Prize winning algorithm:

Mixer and Predictor: Mixers now skip weight updates when errors are below a certain threshold, which enhances processing speed.

Single Pass Wikipedia Transform: This update reduces the time and disk usage needed for processing large datasets like Wikipedia by simplifying the transformation process from a previous multi-step approach to a single pass, thereby significantly speeding up preprocessing stages.

New Stemming and Context Methods: Utilizing Natural Language Processing techniques like new word types in stemming processes to create more compact and relevant word streams. This not only improves the quality of the training data but also enhances compression, reducing the storage requirements.

Efficient Article Ordering: By embedding entire articles

/r/MachineLearning
https://redd.it/1g1l725

Tailwindcss or bootstrap in 2024 ?

I like tailwindcss but once I spend two weeks without using it I forget a lot of class names. Also I end up just copy pasting whatever free design I get which is not really deep learning. But I do hate how much it makes my html pages clutered, especially since I mostly build server rendered websites.
Bootstrap seems fine, with way less classes to put on the html so, I hesitate to double down on bootstrap

/r/flask
https://redd.it/1g0408h

How to handle file uploads with Flask, PostgreSQL, and markdown with restricted access?

I'm working on a Flask application where users can upload various types of files. Here are some examples of how file uploads are used:

Users upload profile pictures or company logos.
When creating a job, users can write a denoscription in markdown that includes image/file uploads for visualization.
Jobs have "Items" (similar to jobs), which also include markdown denoscriptions with image/file uploads.
Users can comment on Jobs and Items, and the comments can include multiple images/files.

Each of these objects corresponds to a PostgreSQL model (User, Company, Job, Item, Comment), and each model already has CRUD APIs and an authorization map. For example, only users in a company can see Jobs related to that company.

# My Requirements:

1. I want to handle file uploads and display them properly in markdown (like pasting an image and getting a URL).
2. I need to restrict access to image URLs based on the same authorization map as the object that owns the image (e.g., images in a comment should only be visible to authorized users).
3. Images/files should "live" with the object that references them. If the object (e.g., a comment) is deleted, the associated images/files should also be deleted.

# Example Flow:

1. A user starts writing a

/r/flask
https://redd.it/1g00toj