[Elphel-support] datarate issue - HDD recording, while streaming.

Tue Oct 18 22:23:25 PDT 2011

Thankyou to Andrey, Oleg and Sebastian-

These developments are very assuring!

Not only is there the immediate solution of reducing php calls, but-
Combined with the proposed fix to the streamer will enable very high
datarates to be achieved!

This is exciting :)

On Tue, Oct 18, 2011 at 6:25 AM, Sebastian Pichelhofer <
sebastian.pichelhofer at gmail.com> wrote:

> good idea!
>
> On Mon, Oct 17, 2011 at 21:24, Andrey Filippov
> <support-list at support.elphel.com> wrote:
> > Sebastian, nice job!
> >
> > You may try to adjust coring value - that will allow you to change file
> size
> > more gradually than in 1% increments.
> >
> > Andrey
> >
> > On Mon, Oct 17, 2011 at 1:14 PM, Sebastian Pichelhofer
> > <sebastian.pichelhofer at gmail.com> wrote:
> >>
> >> I ran a full measurement cycle.
> >>
> >> With and without streamer enabled, I also reduced the AJAX call
> >> frequency in camogmgui to once every 5 seconds to not give PHP an
> >> additional CPU load in the camera. All values are averaged over 5
> >> seconds.
> >>
> >> JPEG Quality and data-rate are relative not absolute as the data-rate
> >> depends on the actual image content but I kept it as reference during
> >> the tests.
> >>
> >> All tests were done with 1920x1088 at 25fps RGB Color mode.
> >>
> >> Regards Sebastian
> >>
> >> On Mon, Oct 17, 2011 at 18:12, Oleg K Dzhimiev <oleg at elphel.com> wrote:
> >> > Nathan,
> >> > I was able to reduce the PHP's cpu usage and could run recording
> >> > 1920x1088 at 25fps, quality 93, with streamer working, single frame size
> >> > ~300K-~400K,
> >> > streamer ~60% cpu, camogm ~30% cpu (same w/o streamer).
> >> > The problem is that the php calls (status requests) are "too frequent"
> >> > so
> >> > that they eat ~30%. I used my own interface (originally for our
> panorama
> >> > camera) for recording where the status calls interval can be
> programmed.
> >> > So that interval was 5 sec - can be more frequent, 2 sec worked also.
> >> > It could be a solution if Sebastian implements it in his GUI.
> >> > Regards,
> >> > Oleg
> >> >
> >> > On 11 October 2011 22:24, Nathan Clark <nathan at nathanclark.com.au>
> >> > wrote:
> >> >>
> >> >> Forwarding a short discussion regarding datarate limitations
> recording
> >> >> to
> >> >> HDD while running streamer simultaneously. Continuing discussion here
> >> >> so
> >> >> that others may benefit.
> >> >> Problem:
> >> >> Cannot record to HDD with high datarates when streamer is running.
> >> >> Cause:
> >> >> Streamer running with high datarate results in high CPU usage causing
> >> >> bottleneck.
> >> >> Fix:
> >> >> • Reduce CPU requirements of webserver
> >> >> • Reduce CPU requirements of streamer
> >> >> Is it possible to have the streamer skip every n frames?
> >> >> This would potentially halve the datarate (if skipping every 2nd
> frame)
> >> >> and as such significantly lower the cpu requirement.
> >> >> The reduced temporal resolution of the stream would still be okay for
> >> >> viewing in many circumstances.
> >> >> Oleg,
> >> >> • I get a similar cpu% usage for php instances.
> >> >> • I do need the use of the streamer, while recording to SSD over
> sata.
> >> >>
> >> >> Cheers,
> >> >> Nathan
> >> >> ---------- Forwarded message ----------
> >> >> From: Oleg K Dzhimiev
> >> >>
> >> >> Also, can we move the discussion to our mailing list.
> >> >> Thanks
> >> >> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
> >> >>>
> >> >>> Nathan,
> >> >>> What's the CPU usage for php while recording? I got ~(24+27)% from
> two
> >> >>> php instances. That might be causing the problem - too many php
> calls
> >> >>> from
> >> >>> the GUI.
> >> >>> In your application do you use the streamer while recording?
> >> >>> Best regards,
> >> >>> Oleg
> >> >>>
> >> >>> On 11 October 2011 05:55, Nathan Clark wrote:
> >> >>>>
> >> >>>> As Sebastian advised, I tested the CPU with streamer enabled...
> >> >>>> We have a problem!
> >> >>>>
> >> >>>> CPU usage for streamer is up at 67%
> >> >>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
> >> >>>>
> >> >>>> The CPU usage directly correlates with data rate, reducing quality
> to
> >> >>>> 20
> >> >>>> (0.86MB/s) results in streamer CPU usage between 3-9%
> >> >>>>
> >> >>>> It looks like the streamer is causing this recording issue by
> >> >>>> bottlenecking the CPU and thus causing the buffer to overrun...
> >> >>>>
> >> >>>> But how to circumvent this issue?
> >> >>>>
> >> >>>>
> >> >>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
> >> >>>>
> >> >>>> > Hi Oleg, I suggested we continue this discussion via email.
> >> >>>> >
> >> >>>> > Below is Nathans original PM to me:
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Hey Sebastian,
> >> >>>> >
> >> >>>> > As time escapes me, its getting closer and closer to Winnie
> >> >>>> > shooting
> >> >>>> > her experimental film for her stereoscopic PHD research...
> >> >>>> > I am not ready :|
> >> >>>> >
> >> >>>> > With problems with my lenses and this sata datarate pain, it's
> >> >>>> > becoming quite scary :shock:
> >> >>>> >
> >> >>>> > But there is always times of stress with all projects! And
> >> >>>> > ultimately,
> >> >>>> > I just remind myself how amazing what we have created really is!
> :D
> >> >>>> >
> >> >>>> > ----------------
> >> >>>> >
> >> >>>> > Did you have time to verify the external HDD recording datarate
> >> >>>> > limits
> >> >>>> > we talked about some time ago?
> >> >>>> >
> >> >>>> > Regards Sebastian
> >> >>>>
> >> >>
> >> >> _______________________________________________
> >> >> Support-list mailing list
> >> >> Support-list at support.elphel.com
> >> >>
> >> >>
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Support-list mailing list
> >> > Support-list at support.elphel.com
> >> >
> >> >
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >> >
> >> >
> >>
> >> _______________________________________________
> >> Support-list mailing list
> >> Support-list at support.elphel.com
> >>
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >>
> >
> >
> > _______________________________________________
> > Support-list mailing list
> > Support-list at support.elphel.com
> >
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >
> >
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20111019/52250832/attachment-0002.html>