Keeping large data available to all users in memory

Not quite sure if this is the correct subreddit for questions like this, but here I go:

I'm currently building a website where I each day want yesterday's financial data cached in memory for the full day, to be quickly available to all users (note that this data is incommon for all users, and can be quite big at ~50 MB). I also want to cache older data on request, e.g. if a user looks at data for some date in 2023, I also want to cache that, although for a shorter time (say maybe 1 h).

Being new to Django, I've tried to read up on the different caching solutions that Django offers, but I'm a bit confused as to which would server me well.

* Does Django offer an existing solution fit for this purpose?
* Would it be easier to set up a manual pure python caching process, which fetches cached data if it exists, and otherwise retrieves and caches it for a specfied time period?
* Can this be solved by simply attaching the data to a middleware instance? Not quite sure how long such an instance lives to provide the attached data.

Thank you!

/r/django
https://redd.it/1guu1t8

Is this a good production setup?

I've just finished deploying my project on Digital Ocean and would like to hear your opinion on the way I've done it and if this is a production-friendly setup. Any feedback is welcome but please be nice as this is my first run-in with this amount of devops.

The application is a bookings management SaaS for small businesses to manage (calendar view) and sell (online store) their experiences (Surf camps, Yoga camps, Wellness retreats, etc)

The stack is a Django app with AlpineJs for frontend all within the same project + Postgres + Redis + Celery.

I set up my Django project with cookiecutter-django from the two scoops guys and for local dev I'm using Docker.


So for local Dev I use Docker and run all these services in one container.


On Digital Ocean I decided not to use Docker and to go for their App Platform app (Django app) + Managed Database (Postgres) + Managed Cache Database (Redis) + Spaces (Static) + App (Celery Worker).

The DO enviroment is connected to my Github where when I'm ready to deploy my local changes I push my local changes using git and DO rebuilds and deploys automatically.


The main reason I

/r/django
https://redd.it/1guuq8b

Retro Terminal Chatbot (GLaDOS)

Hey everyone! I’d like to share a Python project I’ve been working on for a while now. It’s a chatbot based on GLaDOS from the Portal games, with a retro-style terminal interface and her iconic voice. I think any portal fans among you will enjoy it.

# What My Project Does

GLaDOS-Terminal is a fully interactive chatbot experience that brings the personality of GLaDOS to life.

Chatting: The responses are generated using the llama3.2:3b model, giving her personality depth and sarcasm that feels true to the original character.
Voice Synthesis: Her iconic voice is recreated using custom fine-tuned Tacotron 2 and HiFi-GAN models that I developed specifically for this project.
UI and Visuals: The terminal is built using pygame combined with a custom moderngl shader, inspired by the retro visuals from the Portal end credits and [this ](https://github.com/Swordfish90/cool-retro-term)project.

# Target Audience

This project is mainly for:

Portal fans who want a chance to talk to GLaDOS (or at least her chatbot counterpart).
Developers or Python enthusiasts who are curious about integrating AI models, custom shaders, and real-time voice synthesis.
Anyone looking for a cool and unique chatbot experience.

Right now, this is more of a showcase or experimental project since it’s somewhat technical to set

/r/Python
https://redd.it/1gv92l4

Wednesday Daily Thread: Beginner questions

# Weekly Thread: Beginner Questions 🐍

Welcome to our Beginner Questions thread! Whether you're new to Python or just looking to clarify some basics, this is the thread for you.

## How it Works:

1. Ask Anything: Feel free to ask any Python-related question. There are no bad questions here!
2. Community Support: Get answers and advice from the community.
3. Resource Sharing: Discover tutorials, articles, and beginner-friendly resources.

## Guidelines:

This thread is specifically for beginner questions. For more advanced queries, check out our [Advanced Questions Thread](#advanced-questions-thread-link).

## Recommended Resources:

If you don't receive a response, consider exploring r/LearnPython or join the Python Discord Server for quicker assistance.

## Example Questions:

1. What is the difference between a list and a tuple?
2. How do I read a CSV file in Python?
3. What are Python decorators and how do I use them?
4. How do I install a Python package using pip?
5. What is a virtual environment and why should I use one?

Let's help each other learn Python! 🌟

/r/Python
https://redd.it/1gvbziu

Has anyone here built a profitable side project with Django or created one for a client that generates profit?

Hi everyone,

I’m curious to hear from those who’ve created side projects using Django. Have any of you built something that turned out to be profitable, either as a personal project or for a client?

I’m working on a side project myself using Django and DRF, mainly focusing on the backend. While I enjoy the process, I’m also wondering about the potential for turning it into something financially viable.

Thanks in advance for sharing your experiences! I’m hoping this inspires ideas and helps me (and others) approach these projects with a more practical perspective.


/r/django
https://redd.it/1gv7q2v

I'm stucking in installing numpy

Hello everyone , im trying to install numpy library on my terminal and always i face a problem on installing backend dependencies, i try to solve this problem by adding cmake to my system path and the c/cpp compiler and the problem continue , then i try installing .whl files of numpy and still the same problem , can anyone here have a solution for this problem .

/r/Python
https://redd.it/1gvkpbm

I am thinking of making a tool noscript that makes it easier for linux admins

To many commands is a pain to recall the switches and so on. Anyone know of any shell noscript that can provide a menu then sub-menu of options?

/r/Python
https://redd.it/1gvhb2h

Created an AI Research Assistant that actually DOES research! one query FULL document of knowledge!

Automated-AI-Web-Researcher: After months of work, I've made a python program that turns local LLMs running on Ollama into online researchers for you, Literally type a single question or topic and wait until you come back to a text document full of research content with links to the sources and a summary and ask it questions too! and more!



What My Project Does:



This automated researcher uses internet searching and web scraping to gather information, based on your topic or question of choice, it will generate focus areas relating to your topic designed to explore various aspects of your topic and investigate various related aspects of your topic or question to retrieve relevant information through online research to respond to your topic or question. The LLM breaks down your query into up to 5 specific research focuses, prioritising them based on relevance, then systematically investigates each one through targeted web searches and content analysis starting with the most relevant.

Then after gathering the content from those searching and exhausting all of the focus areas, it will then review the content and use the information within to generate new focus areas, and in the past it has often finding new, relevant focus areas based on

/r/Python
https://redd.it/1gvlzax

Boiler plate for flask back end.

I am looking for Flask boiler plate for creating backend services. I am exploring cookie cutter but it seems old. Is any other updated resource available ?

/r/Python
https://redd.it/1gvjp8p

Django Rest Framework Multiple Instance Creation tip

/r/django
https://redd.it/1gvnosg

Migrating from black and flake8 to ruff

as the noscript says, so i'm currently working on a relatively huge python/django codebase, built over the course of 6 years, which has been using black and flake8 for formatting and linting in pre-commit hook, both have their versions unupdated for about 3 years, now i have a somewhat difficult task on hand.

the formatting and linting engine is to be moved to ruff but in such a way that the formatting and linting changes reflected in codebase due to ruff are minimal, i can't seem to figure out a way of exporting either configs from black and flake8 in their current state so i can somehow replicate them in ruff to control the changes due to formatting. if anyone has been in a similar situation or know any potential way i can approach this, that would greatly help. cheers!

pre-commit-config.yaml (in its current state, as you can see versions are a bit older)

repos:
- repo: https://github.com/psf/black
rev: 19.10b0
hooks:
- id: black


/r/Python
https://redd.it/1gvnfvi

moka-py: A high performance caching library for Python written in Rust with TTL/TTI support

Hello!

I'm exited to share my first Rust lib for Python — [moka-py](https://github.com/deliro/moka-py)!

# What My Project Does

**moka-py** is a Python binding for the highly efficient [Moka](https://github.com/moka-rs/moka) caching library written in Rust. This library allows you to leverage the power of Moka's high-performance, feature-rich cache in your Python projects.

# Key Features:

* **Synchronous Cache:** Supports thread-safe, in-memory caching for Python applications.
* **TTL Support:** Automatically evicts entries after a configurable time-to-live (TTL).
* **TTI Support:** Automatically evicts entries after a configurable time-to-idle (TTI).
* **Size-based Eviction:** Automatically removes items when the cache exceeds its size limit using the TinyLFU policy.
* **Concurrency:** Optimized for high-performance, concurrent access in multi-threaded environments.
* **Fully typed:** mypy/pyright friendly. Even decorators

# Example (`@lru_cache` drop-in replacement but with TTL and TTI support):

```
from time import sleep
from moka_py import cached


@cached(maxsize=1024, ttl=10.0, tti=1.0)
def f(x, y):
print("hard computations")
return x + y


f(1, 2) # calls computations
f(1, 2) # gets from the cache
sleep(1.1)
f(1, 2) # calls computations (since TTI has passed)
```

# One more example:

```
from time import sleep
from moka_py import Moka


# Create a cache with a capacity of 100 entries, with a TTL of 30 seconds
# and a TTI of 5.2 seconds. Entries are always removed after 30 seconds
# and are removed after 5.2 seconds if there are

/r/Python
https://redd.it/1gvnsoh

Running 24/7 chromedriver python noscript

Hello guys,
I was wondering if its possible to run a python noscript using a chromedriver on a AWS noscript service, or do I need to use a VPS ? This is simple task, no huge traitment.
Thanks !

/r/Python
https://redd.it/1gvq8z2

Multiple domains and allowedhosts

I built a landing page app with different domain for each page, stored in a model with domain as a field. Is there a way to dynamically create ALLOWED\HOSTS in settings? I have tried this and it doesn't work because models have loaded yet. I want to avoid having a giant list of domains in my settings file and redeploying every time that needs to change. I would rather add LandingPages in the django admin and have it update allowed hosts automatically.

    ALLOWEDHOSTS = list(LandingPage.objects.valueslist('domain', flat=True))

/r/django
https://redd.it/1gvth59

Spelunking in Comments and Documentation for Security Footguns

Hi everyone, we just posted a new article on interesting security footguns that could pop up in applications using third-party Elixir, Python, and Golang libraries. It's a fast read, so check it out! https://blog.includesecurity.com/2024/11/spelunking-in-comments-and-documentation-for-security-footguns/

/r/Python
https://redd.it/1gvxlrz

[D] PhD in RL/ML Theory or LLM

Hi guys,

I'm at a crossroads in my academic journey and would appreciate the community's insights. I'm trying to decide between pursuing a PhD focused on reinforcement learning/ML theory versus specializing in large language models with more experimental/applied research (these are the only two offers I had).

# Key considerations are the following:

# Research Impact

* RL/ML Theory: Foundational work that could advance the field's mathematical understanding
* LLMs: Direct applications in today's most transformative AI systems

# Job Prospects

* Theory: Academia, research labs, potentially more limited industry roles
* LLMs: High industry demand, active research area in both academia and industry

# Long-term Relevance

* Theory: Core principles likely to remain valuable regardless of specific technologies
* LLMs: Currently revolutionary but uncertain long-term trajectory

Personal background

* I'm an international student and about to finish my master program in US, so I no longer has enough time before making the final decision. I used to research in ml theory, but did not end up with a real top conference publication in theory. I personally doubt if I have enough mathematical background to pursue a successful PhD in this area (e.g., at least publish 2 theory papers a year on ICML/NeurIPS/ICLR/COLT/AISTATS). At the same time, I am personally doubting if theory

/r/MachineLearning
https://redd.it/1gvx8vx

Thursday Daily Thread: Python Careers, Courses, and Furthering Education!

# Weekly Thread: Professional Use, Jobs, and Education 🏢

Welcome to this week's discussion on Python in the professional world! This is your spot to talk about job hunting, career growth, and educational resources in Python. Please note, this thread is not for recruitment.

---

## How it Works:

1. Career Talk: Discuss using Python in your job, or the job market for Python roles.
2. Education Q&A: Ask or answer questions about Python courses, certifications, and educational resources.
3. Workplace Chat: Share your experiences, challenges, or success stories about using Python professionally.

---

## Guidelines:

- This thread is not for recruitment. For job postings, please see r/PythonJobs or the recruitment thread in the sidebar.
- Keep discussions relevant to Python in the professional and educational context.

---

## Example Topics:

1. Career Paths: What kinds of roles are out there for Python developers?
2. Certifications: Are Python certifications worth it?
3. Course Recommendations: Any good advanced Python courses to recommend?
4. Workplace Tools: What Python libraries are indispensable in your professional work?
5. Interview Tips: What types of Python questions are commonly asked in interviews?

---

Let's help each other grow in our careers and education. Happy discussing! 🌟

/r/Python
https://redd.it/1gw2e4u

Just published part 2 of my articles on Python Project Management and Packaging, illustrated with uv


Hey everyone,

Just finished the second part of my comprehensive guide on Python project management. This part covers both building packages and publishing.

It's like the first article, the goal is to dig in the PEPs and specifications to understand what the standard is, why it came to be and how. This is was mostly covered in the build system section of the article.

The article: https://reinforcedknowledge.com/a-comprehensive-guide-to-python-project-management-and-packaging-concepts-illustrated-with-uv-part-2/

I have tried to implement some of your feedback. I worked a lot on the typos (I believe there aren't any but I may be wrong), and I tried to divide the article into three smaller articles:
- Just the high level overview: https://reinforcedknowledge.com/a-comprehensive-guide-to-python-project-management-and-packaging-part-2-high-level-overview/
- The deeper dive into the PEPs and specs for build systems: https://reinforcedknowledge.com/a-comprehensive-guide-to-python-project-management-and-packaging-part-2-source-trees-and-build-systems-interface/
- The deeper dive into PEPs and specs for package formats: https://reinforcedknowledge.com/a-comprehensive-guide-to-python-project-management-and-packaging-part-2-sdists-and-wheels/

In the parent article there are also two smalls sections about uv build and uv publish. I don't think they deserve to be in a separate smaller article and I included them for completeness but anyone can just go uv help <command> and read about the command and it'd be much better. I did explain some small details that I believe that not everyone knows but I don't think it replaces your

/r/Python
https://redd.it/1gw1fe6

Should I Learn a New Tech or Start Applying?

Hello folks,

I've been working with Django for the past 3 months and have hands-on experience in Machine Learning, Computer Vision, and other AI-related projects. I'm pretty confident in Python and have completed two remote internships, each lasting 2 months.

I'm aiming for a decent package of around 5-6 LPA, but I'm at a crossroads:
1)Should I learn a different technology (like Node.js, since many job postings mention it), or is Django enough?
2)Should I start applying for jobs now or focus on adding more skills to match industry demands?

Also, can you suggest platforms or places where I can find Django-related job opportunities? Most openings I come across seem to require JavaScript or Node.js expertise.

Thank you in advance for your advice!

/r/django
https://redd.it/1gw81a7

Offering 50 free places on my Python Udemy course

Thank you to everyone who beta tested my new Udemy course "The 10-Day Python Bootcamp for Engineers and Scientists".

Things are going well and I'm making some income! Given there are literally millions of people out there interested in Python, I figured it wouldn't do any harm to hand out some more vouchers to this community.

As such, here is a link to 50 free vouchers for the course: https://www.udemy.com/course/python-for-engineers-scientists-and-analysts/?couponCode=THANKYOUREDDIT

As always, I'm grateful for your feedback. Enjoy the course if you do take a voucher.

/r/Python
https://redd.it/1gw30h9

Tips to morph Internal DRF App into Multi Tenant SaaS Setup

Dear community,
I learned A TON over the last months from all the posts and great answers here.

My team and I are transforming our Django-React application (utilizing DRF, PostgreSQL, Redis, Celery, and MinIO) into a multi-tenant SaaS platform. Currently, the app relies heavily on manual admin management. Our goal is to streamline client onboarding and empower tenant admins to manage their settings, data, and users independently.

Points where we catch strays :

1. Multi-Tenancy Design: We're debating between using separate schemas or databases. We aim for robust data isolation but are concerned about the complexities of schema migrations and managing multiple databases.
2. Customization: Tenants wish to define custom fields (likely using JSON) and workflows. How can we efficiently handle queries and searches across these fields without overloading PostgreSQL?
3. Global Search: Tenants require robust search functionality, including full-text and nested searches. While Elasticsearch seems suitable, syncing tenant data dynamically poses challenges. In the interim, could PostgreSQL Views facilitate searches across models and relationships?
4. Throttling & Performance: How can we prevent resource hogging (e.g., database queries, Celery tasks) by "noisy neighbors" without complicating resource allocation?
5. Auth/Compliance: Supporting SSO and maintaining tenant-specific audit logs is becoming complex. Any advice on keeping this

/r/django
https://redd.it/1gwaw56