Maximum recursion depth exceeded while calling a python object как исправить

I’ve built a crawler that had to run on about 5M pages (by increasing the url ID) and then parses the pages which contain the info’ I need.

after using an algorithm which run on the urls (200K) and saved the good and bad results I found that the I’m wasting a lot of time. I could see that there are a a few returning subtrahends which I can use to check the next valid url.

you can see the subtrahends quite fast (a little ex’ of the few first “good IDs”) –

510000011 # +8
510000029 # +18
510000037 # +8
510000045 # +8
510000052 # +7
510000060 # +8
510000078 # +18
510000086 # +8
510000094 # +8
510000102 # +8
510000110 # etc'
510000128
510000136
510000144
510000151
510000169
510000177
510000185
510000193
510000201

after crawling about 200K urls which gave me only 14K good results I knew I was wasting my time and need to optimize it, so I run some statistics and built a function that will check the urls while increasing the id with 818178 (top returning subtrahends ) etc’.

this is the function –

def checkNextID(ID):
    global numOfRuns, curRes, lastResult
    while ID < lastResult:
        try:
            numOfRuns += 1
            if numOfRuns % 10 == 0:
                time.sleep(3) # sleep every 10 iterations
            if isValid(ID + 8):
                parseHTML(curRes)
                checkNextID(ID + 8)
                return 0
            if isValid(ID + 18):
                parseHTML(curRes)
                checkNextID(ID + 18)
                return 0
            if isValid(ID + 7):
                parseHTML(curRes)
                checkNextID(ID + 7)
                return 0
            if isValid(ID + 17):
                parseHTML(curRes)
                checkNextID(ID + 17)
                return 0
            if isValid(ID+6):
                parseHTML(curRes)
                checkNextID(ID + 6)
                return 0
            if isValid(ID + 16):
                parseHTML(curRes)
                checkNextID(ID + 16)
                return 0
            else:
                checkNextID(ID + 1)
                return 0
        except Exception, e:
            print "somethin went wrong: " + str(e)

what is basically does is -checkNextID(ID) is getting the first id I know that contain the data minus 8 so the first iteration will match the first “if isValid” clause (isValid(ID + 8) will return True).

lastResult is a variable which saves the last known url id, so we’ll run until numOfRuns is

isValid() is a function that gets an ID + one of the subtrahends and returns True if the url contains what I need and saves a soup object of the url to a global varibale named – ‘curRes‘, it returns False if the url doesn’t contain the data I need.

parseHTML is a function that gets the soup object (curRes), parses the data I need and then saves the data to a csv, then returns True.

if isValid() returns True, we’ll call parseHTML() and then try to check the next ID+the subtrahends (by calling checkNextID(ID + subtrahends), if none of them will return what I’m looking for I’ll increase it with 1 and check again until I’ll find the next valid url.

you can see the rest of the code here

after running the code I got about 950~ good results and suddenly an exception had raised –

“somethin went wrong: maximum recursion depth exceeded while calling a
Python object”

I could see on WireShark that the scipt stuck on id – 510009541 (I started my script with 510000003), the script tried getting the url with that ID a few times before I noticed the error and stopped it.

I was really exciting to see that I got the same results but 25x-40x times faster then my old script, with fewer HTTP requests, it’s very precise, I have missed only 1 result for 1000 good results, which is find by me, it’s impossible to rum 5M times, I had my old script running for 30 hours and got 14-15K results when my new script gave me 960~ results in 5-10 minutes.

I read about stack limitations, but there must be a solution for the algorithm I’m trying to implement in Python (I can’t go back to my old “algorithm”, it will never end).

Thanks!

A Recursive function in programming is a function which calls itself. These functions find applications while constructing programs for factorial, Fibonacci series, Armstrong numbers, etc. The main idea is to break larger programs into smaller, less complex problems. With recursive functions, generating sequences becomes easy. But while using recursive functions, recursionerror may occur in python. In this article, we shall be looking into one such recursionerror: maximum recursion depth exceeded while calling a Python object

What is recursionerror?

As the name suggests, Recursionerror may occur when we are dealing with recursive functions. When we run the recursion function for a large number of times, recursion error is thrown. Python has a limit on the number of times a recursive function can call itself. This is done to ensure that the function does not execute infinitely and stops after some number of iterations. To know the recursion limit in python, we use the following code:

import sys
print(sys.getrecursionlimit())

The output is:

1000

Let us look at an example of RecursionError: maximum recursion depth exceeded. We shall take an example of a factorial function.

The following code shall generate factorial for a given number.

def find_fact(n):
  if n == 0 or n == 1:
    return 1
  else :
    return (n*find_fact(n-1))  

print("Factorial is :", find_fact(5))

Here, this program shall be executed successfully and shall print the below output:

Factorial is : 120

But if we pass a larger number into the find_fact() function, it will throw RecursionError: Maximum Recursion Depth Exceeded error.

print("Factorial is :", find_fact(5000))

Output:

RecursionError: maximum recursion depth exceeded in comparison

Since the recursion function exceeded the limit of 1000 iterations, recursionerror is thrown.

The RecursionError: Maximum Recursion Depth Exceeded error may also be thrown while we are trying to create a nested list whose length exceeds the recursion limit.

Let us take the following example. We have created a function named nested() which accepts one argument – n. Depending on the value of n, the length of that nested list would be created. Let us try to pass a value n greater than the recursion limit.

def nested(n): 
    list1 = list2 = [] 
    for i in range(n): 
        list1.append([])
        list1 = list1[0] 
    return list2

nestedlist = nested(2000)
print(nestedlist)

The output will be a recursion error.

RecursionError: maximum recursion depth exceeded while getting the repr of an object

RecursionError: Maximum Recursion Depth Exceeded While Calling A Python Object

The recursionerror for Maximum Recursion Depth Exceeded While Calling A Python Object is thrown when we are trying to call a python object in Django. The error may also occur while using Flask.

When the interpreter detects that the maximum depth for recursion has reached, it throws the recursionerror. To prevent the stack from getting overflow, python raises the recursionerror.

Best practices to avoid RecursionError: Maximum Recursion Depth Exceeded while calling a Python Object

1. Using other loops instead of recursion

To prevent the error from occurring, we can simply convert the piece of code from recursion to a loop statement.

If we take the example of the factorial function, we can convert it into a non – recursive function. We do that by placing a for loop inside the recursion function. The for loop will execute for a length equal to the value of the factorial number.

def find_fact(n):
  mul = 1
  for i in range(2,n+1):
    mul = mul * i
  return mul

print("Factorial is :", find_fact(1500))

Now, it will not throw any recursion error and simply print the large factorial number.

2. Using sys.setrecursionlimit() function

Else, if we still want to use the recursion function, we can increase the recursion limit from 1000 to a higher number. For that, we have to first import the sys library. Using the sys library, we will use the sys.setrecursionlimit() function.

import sys
sys.setrecursionlimit(2000)

Now, it will not thrown the recursionerror and the program will be executed for larger amount of recursions. On executing the recursive function, it will not throw any error and print its output.

def find_fact(n):
  if n == 0 or n == 1:
    return 1
  else :
    return (n*find_fact(n-1))  

print("Factorial is :", find_fact(1500))

3. Setting boundary conditions

It is necessary to set boundary conditions to ensures that the recursive function comes to an end. In the factorial program, the condition :

'if n == 1 or n == 0 : return 1'

is the boundary condition. It is with this condition that the loop comes to an end.

4. Creating a converging recursion

While writing the recursion condition, one has to ensure that the condition does come to an end and does not continue infinitely. The recursive calls should eventually tend towards the boundary condition.

We have to ensure that we creating a converging condition for that. In the factorial program, the ‘n*fact(n-1)’ is a converging condition that converges the value from n to 1.

5. Using Memoization

We can also use memoization to reduce the computing time of already calculated values. This way, we can speed up the calculations by remembering past calculations.

When recursive calls are made, then with memoization we can store the previously calculated values instead of unnecessarily calculating them again.

That sums up the article on RecursionError: Maximum Recursion Depth Exceeded While Calling A Python Object. If you have any questions in your mind, don’t forget to let us know in the comments below.

Until next time, Keep Learning!

You might have seen a Python recursion error when running your Python code. Why does this happen? Is there a way to fix this error?

A Python RecursionError exception is raised when the execution of your program exceeds the recursion limit of the Python interpreter. Two ways to address this exception are increasing the Python recursion limit or refactoring your code using iteration instead of recursion.

Let’s go through some examples so you can understand how this works.

The recursion begins!

Let’s create a program to calculate the factorial of a number following the formula below:

n! = n * (n-1) * (n-2) * ... * 1

Write a function called factorial and then use print statements to print the value of the factorial for a few numbers.

def factorial(n):
    if n == 0:
        return 1
    else:
        return n*factorial(n-1) 

This is a recursive function…

A recursive function is a function that calls itself. Recursion is not specific to Python, it’s a concept common to most programming languages.

You can see that in the else statement of the if else we call the factorial function passing n-1 as parameter.

The execution of the function continues until n is equal to 0.

Let’s see what happens when we calculate the factorial for two small numbers:

if __name__ == '__main__': 
    print("The factorial of 4 is: {}".format(factorial(4)))
    print("The factorial of 5 is: {}".format(factorial(5)))

[output]
The factorial of 4 is: 24
The factorial of 5 is: 120 

After checking that __name__ is equal to ‘__main__’ we print the factorial for two numbers.

It’s all good.

But, here is what happens if we calculate the factorial of 1000…

print("The factorial of 1000 is: {}".format(factorial(1000)))

[output]
Traceback (most recent call last):
  File "recursion_error.py", line 9, in <module>
    print("The factorial of 1000 is: {}".format(factorial(1000)))
  File "recursion_error.py", line 5, in factorial
    return n*factorial(n-1)
  File "recursion_error.py", line 5, in factorial
    return n*factorial(n-1)
  File "recursion_error.py", line 5, in factorial
    return n*factorial(n-1)
  [Previous line repeated 995 more times]
  File "recursion_error.py", line 2, in factorial
    if n <= 1:
RecursionError: maximum recursion depth exceeded in comparison 

The RecursionError occurs because the Python interpreter has exceeded the recursion limit allowed.

The reason why the Python interpreter limits the number of times recursion can be performed is to avoid infinite recursion and hence avoid a stack overflow.

Let’s have a look at how to find out what the recursion limit is in Python and how to update it.

What is the Recursion Limit in Python?

Open the Python shell and use the following code to see the value of the recursion limit for the Python interpreter:

>>> import sys
>>> print(sys.getrecursionlimit())
1000 

Interesting…the limit is 1000.

To increase the recursion limit to 1500 we can add the following lines at the beginning of our program:

import sys
sys.setrecursionlimit(1500)

If you do that and try to calculate again the factorial of 1000 you get a long number back (no more errors).

The factorial of 1000 is: 4023872600770937735437024339230039857193748642107146325437999104299385123986290205920
.......835777939410970027753472000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

That’s good! But…

…this solution could work if like in this case we are very near to the recursion limit and we are pretty confident that our program won’t end up using too much memory on our system.

How to Catch a Python Recursion Error

One possible option to handle the RecursionError exception is by using try except.

It allows to provide a clean message when your application is executed instead of showing an unclear and verbose exception.

Modify the “main” of your program as follows:

if __name__ == '__main__':
    try:
        print("The factorial of 1000 is: {}".format(factorial(1000)))
    except RecursionError as re:
        print("Unable to calculate factorial. Number is too big.") 

Note: before executing the program remember to comment the line we have added in the section before that increases the recursion limit for the Python interpreter.

Now, execute the code…

You will get the following when calculating the factorial for 1000.

$ python recursion_error.py
Unable to calculate factorial. Number is too big. 

Definitely a lot cleaner than the long exception traceback.

Interestingly, if we run our program with Python 2.7 the output is different:

$ python2 recursion_error.py 
Traceback (most recent call last):
  File "recursion_error.py", line 13, in <module>
    except RecursionError as re:
NameError: name 'RecursionError' is not defined 

We get back a NameError exception because the exception of type RecursionError is not defined.

Looking at the Python documentation I can see that the error is caused by the fact that the RecursionError exception was only introduced in Python 3.5:

So, if you are using a version of Python older than 3.5 replace the RecursionError with a RuntimeError.

if __name__ == '__main__':
    try:
        print("The factorial of 1000 is: {}".format(factorial(1000)))
    except RuntimeError as re:
        print("Unable to calculate factorial. Number is too big.") 

In this way our Python application works fine with Python2:

$ python2 recursion_error.py
Unable to calculate factorial. Number is too big. 

How Do You Stop Infinite Recursion in Python?

As we have seen so far, the use of recursion in Python can lead to a recursion error.

How can you prevent infinite recursion from happening? Is that even something we have to worry about in Python?

Firstly, do you think the code we have written to calculate the factorial could cause an infinite recursion?

Let’s look at the function again…

def factorial(n):
    if n == 0:
        return 1
    else:
        return n*factorial(n-1) 

This function cannot cause infinite recursion because the if branch doesn’t make a recursive call. This means that the execution of our function eventually stops.

We will create a very simple recursive function that doesn’t have an branch breaking the recursion…

def recursive_func():
    recursive_func()

recursive_func() 

When you run this program you get back “RecursionError: maximum recursion depth exceeded”.

$ python recursion_error2.py
Traceback (most recent call last):
  File "recursion_error2.py", line 4, in <module>
    recursive_func()
  File "recursion_error2.py", line 2, in recursive_func
    recursive_func()
  File "recursion_error2.py", line 2, in recursive_func
    recursive_func()
  File "recursion_error2.py", line 2, in recursive_func
    recursive_func()
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

So, in theory this program could have caused infinite recursion, in practice this didn’t happen because the recursion depth limit set by the Python interpreter prevents infinite recursion from occurring.

How to Convert a Python Recursion to an Iterative Approach

Using recursion is not the only option possible. An alternative to solve the RecursionError is to use a Python while loop.

We are basically going from recursion to iteration.

def factorial(n):
    factorial = 1

    while n > 0:
        factorial = factorial*n
        n = n - 1

    return factorial

Firstly we set the value of the factorial to 1 and then at each iteration of the while loop we:

• Multiply the latest value of the factorial by n
• Decrease n by 1

The execution of the while loop continues as long as n is greater than 0.

I want to make sure that this implementation of the factorial returns the same results as the implementation that uses recursion.

So, let’s define a Python list that contains a few numbers. Then we will calculate the factorial of each number using both functions and compare the results.

We use a Python for loop to go through each number in the list.

Our program ends as soon as the factorials calculated by the two functions for a given number don’t match.

def factorial(n):
    factorial = 1

    while n > 0:
        factorial = factorial*n
        n = n - 1

    return factorial

def recursive_factorial(n):
    if n == 0:
        return 1
    else:
        return n*factorial(n-1)

numbers = [4, 9, 18, 23, 34, 56, 78, 88, 91, 1000] 

for number in numbers:
    if factorial(number) != recursive_factorial(number):
        print("ERROR: The factorials calculated by the two functions for the number {} do not match.".format(number))

print("SUCCESS: The factorials calculated by the two functions match") 

Let’s run our program and see what we get:

$ python factorial.py
SUCCESS: The factorials calculated by the two functions match 

Great!

Our implementation of the factorial using an iterative approach works well.

Conclusion

In this tutorial we have seen why the RecursionError occurs in Python and how you can fix it.

Two options you have are:

• Increase the value of the recursion limit for the Python interpreter.
• Use iteration instead of recursion.

Which one are you going to use?

Sign Up for Our Newsletters

Subscribe to get notified of the latest articles. We will never spam you. Be a part of our ever-growing community.

By checking this box, you confirm that you have read and are agreeing to our terms of use regarding the storage of the data submitted through this form.

Ismycode |.

Прежде чем прыгать в ошибку, Максимальная глубина рекурсии превышена в сравнении Отказ Сначала понять основы рекурсии и как работает рекурсион в Python.

Что такое рекурсия?

Рекурсия на языке компьютерных языков – это процесс, в котором функция вызывает себя прямо или косвенно, и соответствующая функция называется рекурсивной функцией.

Классический пример рекурсии

Наиболее классическим примером рекурсивного программирования каждый извлек факториал номера. Факториал числа – это Продукт всех положительных целых чисел меньше или равен данному положительному целым числу.

Например, факториал (5) составляет 5 * 4 * 3 * 2 * 1, а факториал (3) составляет 3 * 2 * 1.

Точно так же вы можете использовать рекурсивные во многих других сценариях, таких как Фибоначчи серии , Башня Ханой , Обход деревьев , DFS графа , и т.д.

Как мы уже знаем, рекурсивные функции вызывают сама прямо или косвенно, и во время этого процесса выполнение должно пройти бесконечно.

Python ограничивает количество раз, когда рекурсивная функция может позвонить сам по себе, чтобы убедиться, что она не выполняется бесконечно и вызывает ошибку переполнения стека.

Как проверить максимальную глубину рекурсии в Python?

Вы можете проверить максимальную глубину рекурсии в Python, используя код Sys.getRecursionLimit (). Python не имеет отличной поддержки для рекурсии из-за отсутствия TRE (устранение рекурсионного хвоста). По умолчанию предельный предел рекурсии в Python составляет 1000.

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return(fibonacci(n-1) + fibonacci(n-2))
print(fibonacci(1500))

#Output RecursionError: maximum recursion depth exceeded in comparison

Как вы исправите максимальную глубину рекурсии RecursionError, при вызове объекта Python?

Давайте напишем рекурсивную функцию для расчета серии Fibonacci для данного номера.

Поскольку вы найдете фибоначчи из 1500, а лимит рекурсии по умолчанию в Python является 1000, вы получите ошибку « RecursionError: максимальная глубина рекурсии превышена в сравнении ».

Это может быть исправлено, увеличивая предел рекурсиона в Python, ниже – фрагмент о том, как вы можете увеличить предел рекурсии.

import sys
sys.setrecursionlimit(1500)

Закрытие мыслей

Этот код устанавливает максимальную глубину рекурсии до 1500, и вы даже можете изменить это на более высокий предел. Тем не менее, не рекомендуется выполнять эту операцию, так как ограничение по умолчанию в основном достаточно хорош, и Python не является функциональным языком, а рекурсия хвоста не является особенно эффективной техникой. Переписать алгоритм итеративно, если возможно, в целом, как правило, является лучшей идеей.

Пост Максимальная глубина рекурсии Python превышена в сравнении появился первым на INSMYCODE Отказ

Оригинал: “https://dev.to/itsmycode/python-maximum-recursion-depth-exceeded-in-comparison-4a70”