Stack overflow python как исправить - Сайт, где вы сможете решить свои вопросы

I read on internet similar problem, but none of the answer could help me. I have a function that for each line of data (data have around 2’000’000 rows) do something and then recall the same function with different parameter depending on what it has done. The issue is that after a while I get this error in the terminal: ‘Fatal Python error: Cannot recover from stack overflow.’

It seams that the most frequent mistake causing this error is infinite loop, but I controlled and have no infinite loop. Hence, to me the issue it that ‘sys.getrecursionlimit()’ is set to 3000, which means that after 3000 call of the same function it will give me the error.

The first thing, is that I dont understand the difference between ‘Fatal Python error: Cannot recover from stack overflow.’ in the terminal, or a ‘RecursionError: maximum recursion depth exceeded in comparison’ in the jupyternotebook. Indeed, to me it can come from the same mistake (e.g. infinite loop).

When replacing my function by a simple one called ‘test_’, I have the following code:

import sys
print(sys.getrecursionlimit())

def test_(x,t):
    x = x+1
    if x<t:
        test_(x=x,t=t)

print(test_(0,2971)) # output: None
print(test_(0,2972)) # RecursionError: maximum recursion depth exceeded in comparison

3000

None

————————————————————————— RecursionError Traceback (most recent call
last) in ()
8
9 print(test_(0,2971))
—> 10 print(test_(0,2972))

in test_(x, t)
5 x = x+1
6 if x 7 test_(x=x,t=t)
8
9 print(test_(0,2971))

… last 1 frames repeated, from the frame below …

in test_(x, t)
5 x = x+1
6 if x 7 test_(x=x,t=t)
8
9 print(test_(0,2971))

RecursionError: maximum recursion depth exceeded in comparison

To overcome this issue I adapted the function without loosing the ‘continuity of the run’, so that I can use batches:

for i in np.arange(0,9000,2000):
    test_(i,i+2000)

Would someone have nicer solution? Also, in general its a bad idea to do recursive function when we know that we have a lot of iteration to do? Also does anyone knows how I could print the recursiondeepth at each loop?

I am working on a Linux virtual environement, with jupyter notebook, on python 3.6 with anaconda.

Источник

Stack overflow instead of ‘maximum recursion error’

Usually when you use a function recursively (I think) 1000 times, it will error: “Maximum recursion depth of 999 reached”. However, when I was trying to break my program I got this error instead:

Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fc9f48c9700 (most recent call first):
  File "./player_guesses.py", line 24 in takeGuess
  File "./player_guesses.py", line 27 in takeGuess
  File "./player_guesses.py", line 27 in takeGuess
.
.
.
  File "./player_guesses.py", line 27 in takeGuess
  ...





Aborted (core dumped)

The function that was recursing was this:

#In player_guesses.py
def takeGuess():
    guess = input("Take a guess: ")
    if len(guess) != 3 or not guess.isdigit():
        print("The guess '{}' is not valid, try again!".format(guess))
        return takeGuess()
    else:
        return guess

Why do I get a stack overflow and not a recursion error, as I was expecting?

Edit: I accomplished this by running the program and holding Return, so it kept inputting an empty string.

Edit: I think this is somehow caused by the input(), but why would that cause an overflow?

Edit: Unintended use of Python ahead, but here’s another way to get a stack overflow:

def rec():
    try:
        rec():
    except RecursionError:
        rec()

Источник

In Python, errors are exceptions that occur when the interpreter encounters a problem while running a program. These exceptions can be caught and managed using try-except blocks, allowing a program to handle problems gracefully and recover from them. There are many different types of errors in Python, each with its own meaning and origin. Python memory error is one such error. Let’s take a closer look into the python memory error.

What is Python Memory Error?

A Python memory error occurs when a Python program runs out of memory while attempting to allocate space for new objects. This error usually occurs when a program attempts to generate a massive data structure, such as a list or dictionary, or when it creates little objects continuously until the available memory is consumed.

When a memory error occurs, Python throws a MemoryError exception, indicating that the needed memory could not be allocated. This exception can be handled gracefully by implementing a try-except block, allowing the program to handle any necessary cleanup or error reporting.

Here’s an example of a MemoryError in Python:

# Creating a large list
my_list = [i for i in range(1000000000)]

This code creates a list of one billion integers. Depending on the available memory on your system, this can quickly consume all available memory and raise a MemoryError. The error message might look something like this:

Traceback (most recent call last):
  File "", line 1, in 
MemoryError

To avoid this error, you can use a generator expression instead of a list comprehension, as generators create values on-the-fly and don’t store them in memory all at once. Here’s an example:

# Using a generator expression
my_generator = (i for i in range(1000000000))

This code creates a generator expression that generates the integers from 0 to 999,999,999 on-the-fly, rather than creating a list that stores all the integers in memory. This can help avoid the MemoryError.

Types of Python Memory Error

Here are some more typical ways that Python memory errors might be classified:

1. Out of Memory Error

An Out of Memory Error in Python occurs when a program tries to allocate more memory than is available on the system. This can happen when a user is working with large datasets, images, or other memory-consumption tasks. For example, suppose a user tries to create a list of one billion integers on a system with limited memory, it may run out of memory and throws an Out of Memory Error.

2. Stack Overflow Error

A Stack Overflow Error occurs when a program’s call stack becomes too deep, causing a stack overflow. This can happen when using a recursive function. For example, if a program uses a recursive function with no base case, it may run out of stack space and raise a Stack Overflow Error.

3. Memory Leak Error

A Memory Leak Error occurs when a program continuously allocates memory without releasing it, leading to a gradual increase in memory usage over time. This can happen when a program repeatedly allocates memory to new objects or data structures without properly releasing old memory, or when there are circular references between objects that prevent them from being garbage-collected.

Ways to Handle Python Memory Error

These are some ways to handle Python memory errors:

1. Appropriate Python Set-up

This simplest, but perhaps least intuitive, answer to a MemoryError is actually related to a potential problem with your Python setup. If you install the 32-bit version of Python on a 64-bit system, you will have highly limited access to the system’s RAM. This restricted access may result in MemoryErrors on programs that your computer would normally be able to handle.

2. Use Generators or Iterators

Instead of putting all of the data into memory at once, use generators or iterators to process data in smaller chunks. This can help to reduce memory use and avoid memory faults. For example, in a function, use yield to generate data one item at a time.

def generate_data():
    for i in range(1000000):
        yield i

for data in generate_data():
    process_data(data)

3. Use a Database or External Storage

If the data set is too large to fit in memory, consider storing it in a database or an file. This allows the program to access the data without needing to load it all into memory all at once. Here’s an example using the pandas library to read a large CSV file:

# Importing the pandas library
import pandas as pd

# Reading a large CSV file
df = pd.read_csv('large_file.csv')

4. Improve Memory Consumption

Review the code and look for places where memory usage can be improved. For example, instead of copying lists, use slicing, map, or filter built-in functions instead of loops.

5. Catch and Handle Exceptions

Catch MemoryError exceptions and handle them in a way that allows the program to continue functioning. For example, log the error and try again later, or display an error message to the user and politely exit the program. Here’s an code snippet to demonstrate this:

try:
    # Code that might raise a MemoryError
except MemoryError:
    # Handle the error

6. Memory Profiler to Identify Memory Leaks

A memory profiler is a tool that can be used to identify memory leaks in Python programs. Memory leaks occur when a program does not release memory that is no longer needed, causing memory usage to increase over time. Memory profilers help identify the parts of a program that are responsible for memory leaks, allowing developers to address these issues and improve the program’s memory efficiency.

Conclusion
Python memory error occurs when a program tries to use more memory than is available. This can happen due to various reasons, such as not deallocating the memory of unused objects or improper memory usage. To avoid and fix memory errors, it’s important to be aware of the memory usage of our programs and take necessary steps to manage our memory, such as using generators in place of lists, deleting unused objects, and using external storage. By being mindful of memory usage, we can write more efficient Python programs.

FAQs Related to Python Memory Error

Here are some frequently asked questions (FAQs) about Python memory errors:

Q1 – What is a memory error in python?
Ans – A Python memory error occurs when a program tries to allocate more memory than the system can provide. This can happen when a program tries to store too much data in memory or when a program creates too many objects.

Q2 – What causes memory errors in python?
Ans – Python memory errors can be caused by a variety of factors, including a lack of available memory on the system, inefficient code that creates too many objects, or a data set that is too large to fit into memory.

Q3 – How to handle Python memory errors?
Ans – We can handle Python memory errors by catching the MemoryError exception and taking appropriate action, such as freeing up memory or terminating the program. Additionally, we can also use profiling tools to identify memory usage in our code.

Q4 – What should I do if I encounter a Python memory error?
Ans – If you encounter a Python memory error, try to identify the source of the error by using profiling tools or by analyzing your python code. Once you have identified the source, take appropriate action to reduce memory usage, such as using memory-efficient data structures or deleting objects that are no longer needed. If you are working with large data sets, consider using external storage to store them.

Источник

Добрый день! Очень прошу помочь разобраться с проблемой.
Написал бота, запускаю все работает без проблем одна и та же функция постоянно повторяется, но спустя 40 минут бот вылетает с такой ошибкой

Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Current thread 0x00007f81917be700 (most recent call first):
python ine 40 in Orders

Orders – это и есть функция которая при запуске работает без проблем

Сам код функции

def Orders(Operation):
    try:
        url = ''.join((serv, orders)) 
        getOrders = requests.get(url=url, headers= headers)
        MyOrders = dict.copy((getOrders.json()))
        if MyOrders['payload'] == [] and Operation == 'Sell' or MyOrders['payload'] == [] and Operation == 'None': 
            time.sleep(1)
            provPortf() 
        elif MyOrders['payload'] == [] and Operation == 'Buy':
            time.sleep(1)
            MarketBuy()
        else:
            time.sleep(2)
            Orders(Operation)
    except Exception as e:
        err = 'err'
        time.sleep(2)
        Orders(Operation)

И вот я бы понял если бы что то менялось, и какая то другая функция срабатывала, но получается так, что в течении 40 минут функция orders вызывает сама себя и работает в бесконечном цикле, и нет никаких проблем, но спустя какое то время получаю ошибку очень прошу помочь разобраться

Источник

This fix does not help in my case, after applying the above fix, I still get the error message:
Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fa28e7bc700 (most recent call first):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 78 in _worker
  File "/usr/local/lib/python3.7/threading.py", line 870 in run
  File "/usr/local/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/usr/local/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fa28effd700 (most recent call first):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 78 in _worker
  File "/usr/local/lib/python3.7/threading.py", line 870 in run
  File "/usr/local/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/usr/local/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007fa28f7fe700 (most recent call first):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/type_api.py", line 67 in operate
  File "<string>", line 1 in <lambda>
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/operators.py", line 432 in __getitem__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 762 in operate
Update, fixed my issue:
    check_item = crud.item.get_by_user(
        db=db, id=item_id, user_id=current_user.id
    )
    pd_check_item = ItemCheck.from_orm(check_item) # Fix: convert from DB model to pydantic model 
    check_data = jsonable_encoder(pd_check_item, exclude_none=True) # Breaks if accept db model directly
    return check_data
It seems jsonable_encoder can no longer accept a DB model and convert the data, it only accepts a pydantic model.

How to import ItemCheck

Источник

What is Python Memory Error?

Types of Python Memory Error

1. Out of Memory Error

2. Stack Overflow Error

3. Memory Leak Error

Ways to Handle Python Memory Error

1. Appropriate Python Set-up

2. Use Generators or Iterators

3. Use a Database or External Storage

4. Improve Memory Consumption

5. Catch and Handle Exceptions

6. Memory Profiler to Identify Memory Leaks

FAQs Related to Python Memory Error

Вам также может понравиться

Кислый кетчуп как исправить

Как найти актрису в москве

Как найти относительную погрешность по классу точности

Добавить комментарий Отменить ответ