[Elphel-support] datarate issue - HDD recording, while streaming.

[Sebastian Pichelhofer]

good idea!

On Mon, Oct 17, 2011 at 21:24, Andrey Filippov
<support-list at support.elphel.com> wrote:
> Sebastian, nice job!
>
> You may try to adjust coring value - that will allow you to change file size
> more gradually than in 1% increments.
>
> Andrey
>
> On Mon, Oct 17, 2011 at 1:14 PM, Sebastian Pichelhofer
> <sebastian.pichelhofer at gmail.com> wrote:
>>
>> I ran a full measurement cycle.
>>
>> With and without streamer enabled, I also reduced the AJAX call
>> frequency in camogmgui to once every 5 seconds to not give PHP an
>> additional CPU load in the camera. All values are averaged over 5
>> seconds.
>>
>> JPEG Quality and data-rate are relative not absolute as the data-rate
>> depends on the actual image content but I kept it as reference during
>> the tests.
>>
>> All tests were done with 1920x1088 at 25fps RGB Color mode.
>>
>> Regards Sebastian
>>
>> On Mon, Oct 17, 2011 at 18:12, Oleg K Dzhimiev <oleg at elphel.com> wrote:
>> > Nathan,
>> > I was able to reduce the PHP's cpu usage and could run recording
>> > 1920x1088 at 25fps, quality 93, with streamer working, single frame size
>> > ~300K-~400K,
>> > streamer ~60% cpu, camogm ~30% cpu (same w/o streamer).
>> > The problem is that the php calls (status requests) are "too frequent"
>> > so
>> > that they eat ~30%. I used my own interface (originally for our panorama
>> > camera) for recording where the status calls interval can be programmed.
>> > So that interval was 5 sec - can be more frequent, 2 sec worked also.
>> > It could be a solution if Sebastian implements it in his GUI.
>> > Regards,
>> > Oleg
>> >
>> > On 11 October 2011 22:24, Nathan Clark <nathan at nathanclark.com.au>
>> > wrote:
>> >>
>> >> Forwarding a short discussion regarding datarate limitations recording
>> >> to
>> >> HDD while running streamer simultaneously. Continuing discussion here
>> >> so
>> >> that others may benefit.
>> >> Problem:
>> >> Cannot record to HDD with high datarates when streamer is running.
>> >> Cause:
>> >> Streamer running with high datarate results in high CPU usage causing
>> >> bottleneck.
>> >> Fix:
>> >> • Reduce CPU requirements of webserver
>> >> • Reduce CPU requirements of streamer
>> >> Is it possible to have the streamer skip every n frames?
>> >> This would potentially halve the datarate (if skipping every 2nd frame)
>> >> and as such significantly lower the cpu requirement.
>> >> The reduced temporal resolution of the stream would still be okay for
>> >> viewing in many circumstances.
>> >> Oleg,
>> >> • I get a similar cpu% usage for php instances.
>> >> • I do need the use of the streamer, while recording to SSD over sata.
>> >>
>> >> Cheers,
>> >> Nathan
>> >> ---------- Forwarded message ----------
>> >> From: Oleg K Dzhimiev
>> >>
>> >> Also, can we move the discussion to our mailing list.
>> >> Thanks
>> >> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
>> >>>
>> >>> Nathan,
>> >>> What's the CPU usage for php while recording? I got ~(24+27)% from two
>> >>> php instances. That might be causing the problem - too many php calls
>> >>> from
>> >>> the GUI.
>> >>> In your application do you use the streamer while recording?
>> >>> Best regards,
>> >>> Oleg
>> >>>
>> >>> On 11 October 2011 05:55, Nathan Clark wrote:
>> >>>>
>> >>>> As Sebastian advised, I tested the CPU with streamer enabled...
>> >>>> We have a problem!
>> >>>>
>> >>>> CPU usage for streamer is up at 67%
>> >>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
>> >>>>
>> >>>> The CPU usage directly correlates with data rate, reducing quality to
>> >>>> 20
>> >>>> (0.86MB/s) results in streamer CPU usage between 3-9%
>> >>>>
>> >>>> It looks like the streamer is causing this recording issue by
>> >>>> bottlenecking the CPU and thus causing the buffer to overrun...
>> >>>>
>> >>>> But how to circumvent this issue?
>> >>>>
>> >>>>
>> >>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
>> >>>>
>> >>>> > Hi Oleg, I suggested we continue this discussion via email.
>> >>>> >
>> >>>> > Below is Nathans original PM to me:
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Hey Sebastian,
>> >>>> >
>> >>>> > As time escapes me, its getting closer and closer to Winnie
>> >>>> > shooting
>> >>>> > her experimental film for her stereoscopic PHD research...
>> >>>> > I am not ready :|
>> >>>> >
>> >>>> > With problems with my lenses and this sata datarate pain, it's
>> >>>> > becoming quite scary :shock:
>> >>>> >
>> >>>> > But there is always times of stress with all projects! And
>> >>>> > ultimately,
>> >>>> > I just remind myself how amazing what we have created really is! :D
>> >>>> >
>> >>>> > ----------------
>> >>>> >
>> >>>> > Did you have time to verify the external HDD recording datarate
>> >>>> > limits
>> >>>> > we talked about some time ago?
>> >>>> >
>> >>>> > Regards Sebastian
>> >>>>
>> >>
>> >> _______________________________________________
>> >> Support-list mailing list
>> >> Support-list at support.elphel.com
>> >>
>> >> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>> >>
>> >
>> >
>> > _______________________________________________
>> > Support-list mailing list
>> > Support-list at support.elphel.com
>> >
>> > http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>> >
>> >
>>
>> _______________________________________________
>> Support-list mailing list
>> Support-list at support.elphel.com
>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>
>
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
>