[Elphel-support] datarate issue - HDD recording, while streaming.

[Sebastian Pichelhofer]

I created a new wiki page about cammkv, an alternative HDD recorder:
http://wiki.elphel.com/index.php?title=Cammkv

I was wondering if it performed better than camogm since it was coded
from scratch by Andrey Latin.

In my tests I tried to continuously increase datarate until it would
overflow (ssh access only), no streamer or webinterface active.

The maximum throughput I got was around 60Mbit/s. Though it was
difficult to test as the overflow does not occur immediately at higher
datarates but could happen after recording for ~20 seconds. Sometimes
it overflowed sooner sometimes later at the same datarate setting.

Regards Sebastian

On Wed, Oct 12, 2011 at 08:24, Andrey Filippov
<support-list at support.elphel.com> wrote:
> Nathan,
>
> As I remember MPlayer can handle dropped frames nicely, and it will not be
> _many_frames in sequence lost - otherwise the recording will overrun the
> buffer on it's own, without the streamer. "Nice" mode fro the streamer is
> good as it will try to avoid frame drops on recording to the very end,
> increased dropped frames would also indicate that the performance limit is
> close.
>
> Andrey
>
>
> On Tue, Oct 11, 2011 at 10:51 PM, Nathan Clark <nathan at nathanclark.com.au>
> wrote:
>>
>> What would the implications be for the host PC recieving the stream?
>> If the streamer goes to sleep for n frames... Wouldn't the host PC "think"
>> that the connection is lost?
>>
>> On Wed, Oct 12, 2011 at 3:36 PM, Andrey Filippov
>> <support-list at support.elphel.com> wrote:
>>>
>>> Nathan,
>>>
>>> The intended solution would be to tell streamer minimal size of the
>>> camogm  buffer to maintain. And make the streamer "nice" - when the streamer
>>> is awaken by the driver (it happens at each frame, if needed - can sleep for
>>> several frames or until frame #xxx is ready, but it would not take much of
>>> the CPU for the streamer to just check "how is it going" at each frame and
>>> go back to sleep if needed.
>>>
>>> At each frame, streamer can read  the buffer size for camogm  -- driver
>>> will tell that (if camogm is set to update "global read pointer"), compare
>>> to the preset minimum and either stream the current frame or go back to
>>> sleep and skip it.
>>>
>>> All the data structures (accessible over mmap) are already initialized in
>>> the streamer. I think there needs to be just 10 to 20 (with passing the new
>>> parameter - minimal buffer size) additional lines of code - the most time
>>> consuming would be to get inside the code and find the right place - it was
>>> written/modified by several people at different time (even I touched it
>>> once, but was not the last) so it may not be extremely straightforward. But
>>> if somebody will start working on it - I'll try to help/answer the
>>> questions.
>>>
>>> Andrey
>>>
>>>
>>> On Tue, Oct 11, 2011 at 10:24 PM, Nathan Clark
>>> <nathan at nathanclark.com.au> wrote:
>>>>
>>>> Forwarding a short discussion regarding datarate limitations recording
>>>> to HDD while running streamer simultaneously. Continuing discussion here so
>>>> that others may benefit.
>>>> Problem:
>>>> Cannot record to HDD with high datarates when streamer is running.
>>>> Cause:
>>>> Streamer running with high datarate results in high CPU usage causing
>>>> bottleneck.
>>>> Fix:
>>>> • Reduce CPU requirements of webserver
>>>> • Reduce CPU requirements of streamer
>>>> Is it possible to have the streamer skip every n frames?
>>>> This would potentially halve the datarate (if skipping every 2nd frame)
>>>> and as such significantly lower the cpu requirement.
>>>> The reduced temporal resolution of the stream would still be okay for
>>>> viewing in many circumstances.
>>>> Oleg,
>>>> • I get a similar cpu% usage for php instances.
>>>> • I do need the use of the streamer, while recording to SSD over sata.
>>>>
>>>> Cheers,
>>>> Nathan
>>>> ---------- Forwarded message ----------
>>>> From: Oleg K Dzhimiev
>>>>
>>>> Also, can we move the discussion to our mailing list.
>>>> Thanks
>>>> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
>>>>>
>>>>> Nathan,
>>>>> What's the CPU usage for php while recording? I got ~(24+27)% from two
>>>>> php instances. That might be causing the problem - too many php calls from
>>>>> the GUI.
>>>>> In your application do you use the streamer while recording?
>>>>> Best regards,
>>>>> Oleg
>>>>>
>>>>> On 11 October 2011 05:55, Nathan Clark wrote:
>>>>>>
>>>>>> As Sebastian advised, I tested the CPU with streamer enabled...
>>>>>> We have a problem!
>>>>>>
>>>>>> CPU usage for streamer is up at 67%
>>>>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
>>>>>>
>>>>>> The CPU usage directly correlates with data rate, reducing quality to
>>>>>> 20 (0.86MB/s) results in streamer CPU usage between 3-9%
>>>>>>
>>>>>> It looks like the streamer is causing this recording issue by
>>>>>> bottlenecking the CPU and thus causing the buffer to overrun...
>>>>>>
>>>>>> But how to circumvent this issue?
>>>>>>
>>>>>>
>>>>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
>>>>>>
>>>>>> > Hi Oleg, I suggested we continue this discussion via email.
>>>>>> >
>>>>>> > Below is Nathans original PM to me:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Hey Sebastian,
>>>>>> >
>>>>>> > As time escapes me, its getting closer and closer to Winnie shooting
>>>>>> > her experimental film for her stereoscopic PHD research...
>>>>>> > I am not ready :|
>>>>>> >
>>>>>> > With problems with my lenses and this sata datarate pain, it's
>>>>>> > becoming quite scary :shock:
>>>>>> >
>>>>>> > But there is always times of stress with all projects! And
>>>>>> > ultimately,
>>>>>> > I just remind myself how amazing what we have created really is! :D
>>>>>> >
>>>>>> > ----------------
>>>>>> >
>>>>>> > Did you have time to verify the external HDD recording datarate
>>>>>> > limits
>>>>>> > we talked about some time ago?
>>>>>> >
>>>>>> > Regards Sebastian
>>>>>>
>>>>
>>>> _______________________________________________
>>>> Support-list mailing list
>>>> Support-list at support.elphel.com
>>>>
>>>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>>>
>>>
>>
>>
>> _______________________________________________
>> Support-list mailing list
>> Support-list at support.elphel.com
>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>
>
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
>