Very slow download speed from `ObjectDownloadView`

[johnthagen]

I have an ObjectDownloadView that is serving very large files (200MB - 2GB). I've observed that the download speed is very slow, even when pulling downloads over localhost where no actual network is involved at all. I must authenticate the file downloads (not shown the example below), which is why I must use django-downloadview rather than serving them statically.

I cannot use NGINX acceleration due to:

• Document integration of ObjectDownloadView with NGINX acceleration #177

My endpoint looks something like:

class MyModelObjectDownloadView(ObjectDownloadView):
    model_class = MyModel
    file_field = "model"
    basename_field = "filename"

The Model:

class MyModel(models.Model):
    MODEL_UPLOAD_TO_DIR = "models"
    model = models.FileField(upload_to=MODEL_UPLOAD_TO_DIR)
    filename = models.TextField()

URLs:

urlpatterns = [
    ...
    path(
        f"media/{MyModel.MODEL_UPLOAD_TO_DIR}/<int:pk>/",
        MyModelObjectDownloadView.as_view(),
        name=SurfaceModel.IMAGE_UPLOAD_TO_DIR,
    ),
]

I've tested downloading the file using a variety of clients over localhost (also running locally on mac and also within a Linux Docker container) and they all show a similar result:

• httpx
• Chrome
• curl

$ curl http://localhost:8000/media/models/1/ --output out.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  118M  100  118M    0     0  5398k      0  0:00:22  0:00:22 --:--:-- 5663k

This corresponds to about 40Mbps, which seemed very slow for a localhost pull. I also see the python executable running at about 100% CPU, as if it's CPU rather than I/O bound?

Is there something about how django-downloadview streams or chunks the file that contributes to why this is so slow?

Are there any configuration settings to speed up serving files natively from django-downloadview?

[johnthagen]

Interesting that if I read the file into memory and use a vanilla Django HttpResponse directly from a DRF ViewSet, it transfers extremely quickly.

class MyModelViewSet(RetrieveModelMixin, GenericViewSet):
    ...

    def retrieve(self, request: Request, *args: Any, **kwargs: str) -> HttpResponse:
        my_model = self.get_object()
        export_data = Path(my_model.model.path).read_bytes()
        return HttpResponse(
            export_data,
            content_type= "application/octet-stream",
            headers={
                "Content-Disposition": "attachment; "
                'filename="out.bin"',
            },
        )

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  118M  100  118M    0     0   162M      0 --:--:-- --:--:-- --:--:--  161M

I think that is roughly 1.3Gbps. I'm seeing anywhere around a 40x speed improvement.

[johnthagen]

I suspect that the underlying issue is inefficiency in how Django/Python/WSGI stream bytes by looping over the input in the Python interpreter:

django-downloadview/django_downloadview/views/base.py

Line 30 in 338e171

response_class = DownloadResponse

django-downloadview/django_downloadview/response.py

Line 86 in 338e171

class DownloadResponse(StreamingHttpResponse):

django-downloadview/django_downloadview/response.py

Lines 99 to 102 in 338e171

	For large files, or for specialized uses of HTTP streaming,
	applications will usually return an iterator (often a
	generator-iterator) that produces the output in a block-by-block
	fashion.

[johnthagen]

The way I solved this was to simply use pure DRF views to serve files rather than django-downloadview, since I haven't been able to get NGINX acceleration working:

• Document integration of ObjectDownloadView with NGINX acceleration #177

In case this helps others, here is an implementation:

from pathlib import Path
from typing import Any

from django.db.models import Model
from django.forms import FileField
from django.http import HttpResponse
from drf_spectacular.utils import extend_schema, extend_schema_view
from rest_framework.filters import BaseFilterBackend
from rest_framework.mixins import RetrieveModelMixin
from rest_framework.renderers import BaseRenderer
from rest_framework.request import Request
from rest_framework.viewsets import GenericViewSet


class BinaryRenderer(BaseRenderer):
    """A renderer that supports binary file downloads."""

    media_type = "application/octet-stream"
    format = "bin"

    def render(
        self,
        data: bytes,
        accepted_media_type: str | None = None,
        renderer_context: dict[str, Any] | None = None,
    ) -> bytes:
        return data


@extend_schema_view(retrieve=extend_schema(responses=bytes))
class FileDownloadViewSet(RetrieveModelMixin, GenericViewSet):
    """A generic ViewSet that supports serving downloaded files.

    Rather than streaming the file, the entire file is read into memory and sent out, which can
    improve the download speed of files at the cost of maximum memory.

    Fill in the `file_field` and `filename_field` attributes when deriving from this class to set
    which file should be downloaded from the Model.
    """

    filter_backends: list[BaseFilterBackend] = []
    renderer_classes = [BinaryRenderer]

    file_field: str = NotImplemented
    filename_field: str = NotImplemented

    def retrieve(self, request: Request, *args: Any, **kwargs: str) -> HttpResponse:
        model: Model = self.get_object()
        file: FileField = getattr(model, self.file_field)
        file_data = Path(file.path).read_bytes()  # type: ignore[attr-defined]
        file_name: str = getattr(model, self.filename_field)
        return HttpResponse(
            file_data,
            content_type=BinaryRenderer.media_type,
            headers={"Content-Disposition": f'attachment; filename="{file_name}"'},
        )

And an example usage:

class MyModelDownloadViewSet(FileDownloadViewSet):
    queryset = MyModel.objects.all()
    permission_classes = [DjangoObjectPermissions]

    file_field = "file"
    filename_field = "filename"