[Elphel-support] datarate issue - HDD recording, while streaming.

[Andrey Filippov]

Nathan,

As I remember MPlayer can handle dropped frames nicely, and it will not be
_many_frames in sequence lost - otherwise the recording will overrun the
buffer on it's own, without the streamer. "Nice" mode fro the streamer is
good as it will try to avoid frame drops on recording to the very end,
increased dropped frames would also indicate that the performance limit is
close.

Andrey


On Tue, Oct 11, 2011 at 10:51 PM, Nathan Clark <nathan at nathanclark.com.au>wrote:

> What would the implications be for the host PC recieving the stream?
> If the streamer goes to sleep for n frames... Wouldn't the host PC "think"
> that the connection is lost?
>
>
> On Wed, Oct 12, 2011 at 3:36 PM, Andrey Filippov <
> support-list at support.elphel.com> wrote:
>
>> Nathan,
>>
>> The intended solution would be to tell streamer minimal size of the
>> camogm  buffer to maintain. And make the streamer "nice" - when the streamer
>> is awaken by the driver (it happens at each frame, if needed - can sleep for
>> several frames or until frame #xxx is ready, but it would not take much of
>> the CPU for the streamer to just check "how is it going" at each frame and
>> go back to sleep if needed.
>>
>> At each frame, streamer can read  the buffer size for camogm  -- driver
>> will tell that (if camogm is set to update "global read pointer"), compare
>> to the preset minimum and either stream the current frame or go back to
>> sleep and skip it.
>>
>> All the data structures (accessible over mmap) are already initialized in
>> the streamer. I think there needs to be just 10 to 20 (with passing the new
>> parameter - minimal buffer size) additional lines of code - the most time
>> consuming would be to get inside the code and find the right place - it was
>> written/modified by several people at different time (even I touched it
>> once, but was not the last) so it may not be extremely straightforward. But
>> if somebody will start working on it - I'll try to help/answer the
>> questions.
>>
>> Andrey
>>
>>
>> On Tue, Oct 11, 2011 at 10:24 PM, Nathan Clark <nathan at nathanclark.com.au
>> > wrote:
>>
>>> Forwarding a short discussion regarding datarate limitations recording to
>>> HDD while running streamer simultaneously. Continuing discussion here so
>>> that others may benefit.
>>>
>>> *Problem: *
>>> Cannot record to HDD with high datarates when streamer is running.
>>>
>>> *Cause:*
>>> Streamer running with high datarate results in high CPU usage causing
>>> bottleneck.
>>>
>>> *Fix:*
>>> • Reduce CPU requirements of webserver
>>> • Reduce CPU requirements of streamer
>>>
>>> *Is it possible to have the streamer skip every n frames?*
>>> This would potentially halve the datarate (if skipping every 2nd frame)
>>> and as such significantly lower the cpu requirement.
>>> The reduced temporal resolution of the stream would still be okay for
>>> viewing in many circumstances.
>>>
>>> Oleg,
>>>
>>> • I get a similar cpu% usage for php instances.
>>> • I do need the use of the streamer, while recording to SSD over sata.
>>>
>>> Cheers,
>>>
>>> Nathan
>>>
>>> ---------- Forwarded message ----------
>>> From: Oleg K Dzhimiev
>>>
>>> Also, can we move the discussion to our mailing list.
>>> Thanks
>>>
>>> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
>>>
>>>> Nathan,
>>>>
>>>> What's the CPU usage for php while recording? I got ~(24+27)% from two
>>>> php instances. That might be causing the problem - too many php calls from
>>>> the GUI.
>>>>
>>>> In your application do you use the streamer while recording?
>>>>
>>>> Best regards,
>>>> Oleg
>>>>
>>>>
>>>> On 11 October 2011 05:55, Nathan Clark wrote:
>>>>
>>>>> As Sebastian advised, I tested the CPU with streamer enabled...
>>>>> We have a problem!
>>>>>
>>>>> CPU usage for streamer is up at 67%
>>>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
>>>>>
>>>>> The CPU usage directly correlates with data rate, reducing quality to
>>>>> 20 (0.86MB/s) results in streamer CPU usage between 3-9%
>>>>>
>>>>> It looks like the streamer is causing this recording issue by
>>>>> bottlenecking the CPU and thus causing the buffer to overrun...
>>>>>
>>>>> But how to circumvent this issue?
>>>>>
>>>>>
>>>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
>>>>>
>>>>> > Hi Oleg, I suggested we continue this discussion via email.
>>>>> >
>>>>> > Below is Nathans original PM to me:
>>>>> >
>>>>> >
>>>>> >
>>>>> > Hey Sebastian,
>>>>> >
>>>>> > As time escapes me, its getting closer and closer to Winnie shooting
>>>>> > her experimental film for her stereoscopic PHD research...
>>>>> > I am not ready :|
>>>>> >
>>>>> > With problems with my lenses and this sata datarate pain, it's
>>>>> > becoming quite scary :shock:
>>>>> >
>>>>> > But there is always times of stress with all projects! And
>>>>> ultimately,
>>>>> > I just remind myself how amazing what we have created really is! :D
>>>>> >
>>>>> > ----------------
>>>>> >
>>>>> > Did you have time to verify the external HDD recording datarate
>>>>> limits
>>>>> > we talked about some time ago?
>>>>> >
>>>>> > Regards Sebastian
>>>>>
>>>>>
>>> _______________________________________________
>>> Support-list mailing list
>>> Support-list at support.elphel.com
>>>
>>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>>
>>>
>>
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20111012/8c2f9e3d/attachment-0002.html>