[Elphel-support] datarate issue - HDD recording, while streaming.

[Nathan Clark]

What would the implications be for the host PC recieving the stream?
If the streamer goes to sleep for n frames... Wouldn't the host PC "think"
that the connection is lost?


On Wed, Oct 12, 2011 at 3:36 PM, Andrey Filippov <
support-list at support.elphel.com> wrote:

> Nathan,
>
> The intended solution would be to tell streamer minimal size of the camogm
> buffer to maintain. And make the streamer "nice" - when the streamer is
> awaken by the driver (it happens at each frame, if needed - can sleep for
> several frames or until frame #xxx is ready, but it would not take much of
> the CPU for the streamer to just check "how is it going" at each frame and
> go back to sleep if needed.
>
> At each frame, streamer can read  the buffer size for camogm  -- driver
> will tell that (if camogm is set to update "global read pointer"), compare
> to the preset minimum and either stream the current frame or go back to
> sleep and skip it.
>
> All the data structures (accessible over mmap) are already initialized in
> the streamer. I think there needs to be just 10 to 20 (with passing the new
> parameter - minimal buffer size) additional lines of code - the most time
> consuming would be to get inside the code and find the right place - it was
> written/modified by several people at different time (even I touched it
> once, but was not the last) so it may not be extremely straightforward. But
> if somebody will start working on it - I'll try to help/answer the
> questions.
>
> Andrey
>
>
> On Tue, Oct 11, 2011 at 10:24 PM, Nathan Clark <nathan at nathanclark.com.au>wrote:
>
>> Forwarding a short discussion regarding datarate limitations recording to
>> HDD while running streamer simultaneously. Continuing discussion here so
>> that others may benefit.
>>
>> *Problem: *
>> Cannot record to HDD with high datarates when streamer is running.
>>
>> *Cause:*
>> Streamer running with high datarate results in high CPU usage causing
>> bottleneck.
>>
>> *Fix:*
>> • Reduce CPU requirements of webserver
>> • Reduce CPU requirements of streamer
>>
>> *Is it possible to have the streamer skip every n frames?*
>> This would potentially halve the datarate (if skipping every 2nd frame)
>> and as such significantly lower the cpu requirement.
>> The reduced temporal resolution of the stream would still be okay for
>> viewing in many circumstances.
>>
>> Oleg,
>>
>> • I get a similar cpu% usage for php instances.
>> • I do need the use of the streamer, while recording to SSD over sata.
>>
>> Cheers,
>>
>> Nathan
>>
>> ---------- Forwarded message ----------
>> From: Oleg K Dzhimiev
>>
>> Also, can we move the discussion to our mailing list.
>> Thanks
>>
>> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
>>
>>> Nathan,
>>>
>>> What's the CPU usage for php while recording? I got ~(24+27)% from two
>>> php instances. That might be causing the problem - too many php calls from
>>> the GUI.
>>>
>>> In your application do you use the streamer while recording?
>>>
>>> Best regards,
>>> Oleg
>>>
>>>
>>> On 11 October 2011 05:55, Nathan Clark wrote:
>>>
>>>> As Sebastian advised, I tested the CPU with streamer enabled...
>>>> We have a problem!
>>>>
>>>> CPU usage for streamer is up at 67%
>>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
>>>>
>>>> The CPU usage directly correlates with data rate, reducing quality to 20
>>>> (0.86MB/s) results in streamer CPU usage between 3-9%
>>>>
>>>> It looks like the streamer is causing this recording issue by
>>>> bottlenecking the CPU and thus causing the buffer to overrun...
>>>>
>>>> But how to circumvent this issue?
>>>>
>>>>
>>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
>>>>
>>>> > Hi Oleg, I suggested we continue this discussion via email.
>>>> >
>>>> > Below is Nathans original PM to me:
>>>> >
>>>> >
>>>> >
>>>> > Hey Sebastian,
>>>> >
>>>> > As time escapes me, its getting closer and closer to Winnie shooting
>>>> > her experimental film for her stereoscopic PHD research...
>>>> > I am not ready :|
>>>> >
>>>> > With problems with my lenses and this sata datarate pain, it's
>>>> > becoming quite scary :shock:
>>>> >
>>>> > But there is always times of stress with all projects! And ultimately,
>>>> > I just remind myself how amazing what we have created really is! :D
>>>> >
>>>> > ----------------
>>>> >
>>>> > Did you have time to verify the external HDD recording datarate limits
>>>> > we talked about some time ago?
>>>> >
>>>> > Regards Sebastian
>>>>
>>>>
>> _______________________________________________
>> Support-list mailing list
>> Support-list at support.elphel.com
>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20111012/1cd89fc5/attachment-0002.html>