Deadlock in Blob__write_to_queue/git_filter_list_stream_blob

[Werkov]

Here's rough code that I have (despite I can reproduce the issue at will, the setup is non-trivial and there are some data dependencies, so I present hopefully the relevant parts).

      # my code
      def from_source_git(self, index, repo, sha, name, current_head, move_upstream):
          tree = repo[sha].peel(pygit2.Tree)
          blob = tree[name]                                                                                                                                             
          bfile = pygit2.BlobIO(blob)
          self._from_file(index, bfile, name, current_head, move_upstream)
          print("loaded")                                                                                                                                                                                                                                                        
          bfile.close()
          print("closed")
...
# pygit2 with some debug prints
class _BlobIO(io.RawIOBase):
...
    def close(self) -> None:                                                                                                            
        try:
            self._ready.wait()
            traceback.print_stack()
            print("gonna wait for %s" % self._writer_closed)                                                                            
            # raise Exception()
            self._writer_closed.wait()
            while self._queue is not None and not self._queue.empty():                                                                  
                self._queue.get()                                                                                                       
            self._thread.join()
        except KeyboardInterrupt:                                                                                                       
            pass
        self._queue = None   

So the main thread print this:

loaded
  File "/home/mkoutny/tool", line 159, in <module>
    entry.from_source_git(index, source_repo, sha, name, lib.git_sort.oot, True)
  File "/home/mkoutny/lib.py", line 262, in from_source_git
    bfile.close()
  File "/usr/lib64/python3.13/site-packages/pygit2/blob.py", line 99, in close
    traceback.print_stack()
gonna wait for <threading.Event at 0x7fe3f17a3250: unset>

AFAICS, it waits for the notification from the writer thread, however, that one is waiting on some locks in queue.put (gdb C stacktrace of the relevant part):

#22 0x00007fe3f3093229 in callmethod (tstate=0x55aaa1eeff90, callable=0x7fe37ab30500, format=0x7fe3f25aa56d "y#", va=0x7fe356ffe550) at Objects/call.c:626
#23 PyObject_CallMethod (obj=<optimized out>, name=name@entry=0x7fe3f25aa080 "put", format=format@entry=0x7fe3f25aa56d "y#") at Objects/call.c:645
#24 0x00007fe3f2594d18 in blob_filter_stream_write (s=0x7fe356ffe730, buffer=<optimized out>, len=<optimized out>) at /home/mkoutny/projects/pygit2/src/blob.c:176
#25 0x00007fe3f2483675 in git_filter_list_stream_buffer (filters=<optimized out>,
    buffer=0x7fe35002a150 "[redacted]"..., len=77246,
    target=target@entry=0x7fe356ffe730) at /usr/src/debug/libgit2-1.9.7/src/libgit2/filter.c:1172
#26 0x00007fe3f2489059 in git_filter_list_stream_blob (filters=<optimized out>, blob=<optimized out>, target=target@entry=0x7fe356ffe730) at /usr/src/debug/libgit2-1.9.7/src/libgit2/filter.c:1196
#27 0x00007fe3f2594aef in Blob__write_to_queue (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at /home/mkoutny/projects/pygit2/src/blob.c:361
#28 0x00007fe3f2f72ae2 in cfunction_call (func=0x7fe3e194a660, args=0x7fe37ab30bc0, kwargs=0x7fe37ab30980) at Objects/methodobject.c:539

And there's no progress. (Alas, I'm not familiar with the involved threads/locks to tell who's at fault.) I can provide more details, just ask.

I can observe this even with pygit2 built from git (d532da7) (ehm, I combine that with 1.19 .py files installed on my system).