Bug 103417

Spice server version 0.13.90
Host OS: Ubuntu 16.04
Guest OS: Windows 7

For hardware accelerate, I use GStreamer nvh264enc in place of xh264enc. However, it often get deadlock when GStreamer get restarted. I debugged the gstreamer plugin nvenc and I find the problem in the free_pipeline() and create_pipeline()。 Because there is no mutex lock to protect the process create/free pipeline，when the pipeline status switched in a short time,e.g start -> stop -> start, the nvenc would be in unknown status. In my case, it causes the deadlock inside the nvenc plugin.

I try to fix this bug in a simple way. In the configure_pipeline(), call free_pipeline() before create_pipeline(). This will make sure that we do free the resource of pipeline and it's safe even if the encoder->pipeline is NULL.

The source code is below. Here is the only way to call create_pipeline() except the gstreamer_encoder_new(), but it's like a initialized constructor, so we just take care about the configure_pipeline() which takes responsibility for switching the status of the pipeline.

static gboolean configure_pipeline(SpiceGstEncoder *encoder)
{
    if (!encoder->set_pipeline) {
        return TRUE;
    }

    free_pipeline(encoder->pipeline);

    if (!encoder->pipeline && !create_pipeline(encoder)) {
        return FALSE;
    }
    ...
}


I can't guarantee that it doesn't bring in other bugs. I think there should be a more grace method to fix this problem.