[Elphel-support] datarate issue - HDD recording, while streaming.

Mon Oct 17 08:02:53 PDT 2011

Sebastian,

This program is currently not supported, there were some FPGA changes in the
camera after it was released, so something may break. The purpose of the
project was to have audio support, other than that I do not it may have
performance advantages.

Andrey

On Mon, Oct 17, 2011 at 8:37 AM, Sebastian Pichelhofer <
sebastian.pichelhofer at gmail.com> wrote:

> I created a new wiki page about cammkv, an alternative HDD recorder:
> http://wiki.elphel.com/index.php?title=Cammkv
>
> I was wondering if it performed better than camogm since it was coded
> from scratch by Andrey Latin.
>
> In my tests I tried to continuously increase datarate until it would
> overflow (ssh access only), no streamer or webinterface active.
>
> The maximum throughput I got was around 60Mbit/s. Though it was
> difficult to test as the overflow does not occur immediately at higher
> datarates but could happen after recording for ~20 seconds. Sometimes
> it overflowed sooner sometimes later at the same datarate setting.
>
> Regards Sebastian
>
> On Wed, Oct 12, 2011 at 08:24, Andrey Filippov
> <support-list at support.elphel.com> wrote:
> > Nathan,
> >
> > As I remember MPlayer can handle dropped frames nicely, and it will not
> be
> > _many_frames in sequence lost - otherwise the recording will overrun the
> > buffer on it's own, without the streamer. "Nice" mode fro the streamer is
> > good as it will try to avoid frame drops on recording to the very end,
> > increased dropped frames would also indicate that the performance limit
> is
> > close.
> >
> > Andrey
> >
> >
> > On Tue, Oct 11, 2011 at 10:51 PM, Nathan Clark <
> nathan at nathanclark.com.au>
> > wrote:
> >>
> >> What would the implications be for the host PC recieving the stream?
> >> If the streamer goes to sleep for n frames... Wouldn't the host PC
> "think"
> >> that the connection is lost?
> >>
> >> On Wed, Oct 12, 2011 at 3:36 PM, Andrey Filippov
> >> <support-list at support.elphel.com> wrote:
> >>>
> >>> Nathan,
> >>>
> >>> The intended solution would be to tell streamer minimal size of the
> >>> camogm  buffer to maintain. And make the streamer "nice" - when the
> streamer
> >>> is awaken by the driver (it happens at each frame, if needed - can
> sleep for
> >>> several frames or until frame #xxx is ready, but it would not take much
> of
> >>> the CPU for the streamer to just check "how is it going" at each frame
> and
> >>> go back to sleep if needed.
> >>>
> >>> At each frame, streamer can read  the buffer size for camogm  -- driver
> >>> will tell that (if camogm is set to update "global read pointer"),
> compare
> >>> to the preset minimum and either stream the current frame or go back to
> >>> sleep and skip it.
> >>>
> >>> All the data structures (accessible over mmap) are already initialized
> in
> >>> the streamer. I think there needs to be just 10 to 20 (with passing the
> new
> >>> parameter - minimal buffer size) additional lines of code - the most
> time
> >>> consuming would be to get inside the code and find the right place - it
> was
> >>> written/modified by several people at different time (even I touched it
> >>> once, but was not the last) so it may not be extremely straightforward.
> But
> >>> if somebody will start working on it - I'll try to help/answer the
> >>> questions.
> >>>
> >>> Andrey
> >>>
> >>>
> >>> On Tue, Oct 11, 2011 at 10:24 PM, Nathan Clark
> >>> <nathan at nathanclark.com.au> wrote:
> >>>>
> >>>> Forwarding a short discussion regarding datarate limitations recording
> >>>> to HDD while running streamer simultaneously. Continuing discussion
> here so
> >>>> that others may benefit.
> >>>> Problem:
> >>>> Cannot record to HDD with high datarates when streamer is running.
> >>>> Cause:
> >>>> Streamer running with high datarate results in high CPU usage causing
> >>>> bottleneck.
> >>>> Fix:
> >>>> • Reduce CPU requirements of webserver
> >>>> • Reduce CPU requirements of streamer
> >>>> Is it possible to have the streamer skip every n frames?
> >>>> This would potentially halve the datarate (if skipping every 2nd
> frame)
> >>>> and as such significantly lower the cpu requirement.
> >>>> The reduced temporal resolution of the stream would still be okay for
> >>>> viewing in many circumstances.
> >>>> Oleg,
> >>>> • I get a similar cpu% usage for php instances.
> >>>> • I do need the use of the streamer, while recording to SSD over sata.
> >>>>
> >>>> Cheers,
> >>>> Nathan
> >>>> ---------- Forwarded message ----------
> >>>> From: Oleg K Dzhimiev
> >>>>
> >>>> Also, can we move the discussion to our mailing list.
> >>>> Thanks
> >>>> On 11 October 2011 12:41, Oleg K Dzhimiev wrote:
> >>>>>
> >>>>> Nathan,
> >>>>> What's the CPU usage for php while recording? I got ~(24+27)% from
> two
> >>>>> php instances. That might be causing the problem - too many php calls
> from
> >>>>> the GUI.
> >>>>> In your application do you use the streamer while recording?
> >>>>> Best regards,
> >>>>> Oleg
> >>>>>
> >>>>> On 11 October 2011 05:55, Nathan Clark wrote:
> >>>>>>
> >>>>>> As Sebastian advised, I tested the CPU with streamer enabled...
> >>>>>> We have a problem!
> >>>>>>
> >>>>>> CPU usage for streamer is up at 67%
> >>>>>> (at 1920x1088, rgb, 25fps, quality 90 - data rate 6.9MB/s)
> >>>>>>
> >>>>>> The CPU usage directly correlates with data rate, reducing quality
> to
> >>>>>> 20 (0.86MB/s) results in streamer CPU usage between 3-9%
> >>>>>>
> >>>>>> It looks like the streamer is causing this recording issue by
> >>>>>> bottlenecking the CPU and thus causing the buffer to overrun...
> >>>>>>
> >>>>>> But how to circumvent this issue?
> >>>>>>
> >>>>>>
> >>>>>> On 11/10/2011, at 21:23, Sebastian Pichelhofer wrote:
> >>>>>>
> >>>>>> > Hi Oleg, I suggested we continue this discussion via email.
> >>>>>> >
> >>>>>> > Below is Nathans original PM to me:
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> > Hey Sebastian,
> >>>>>> >
> >>>>>> > As time escapes me, its getting closer and closer to Winnie
> shooting
> >>>>>> > her experimental film for her stereoscopic PHD research...
> >>>>>> > I am not ready :|
> >>>>>> >
> >>>>>> > With problems with my lenses and this sata datarate pain, it's
> >>>>>> > becoming quite scary :shock:
> >>>>>> >
> >>>>>> > But there is always times of stress with all projects! And
> >>>>>> > ultimately,
> >>>>>> > I just remind myself how amazing what we have created really is!
> :D
> >>>>>> >
> >>>>>> > ----------------
> >>>>>> >
> >>>>>> > Did you have time to verify the external HDD recording datarate
> >>>>>> > limits
> >>>>>> > we talked about some time ago?
> >>>>>> >
> >>>>>> > Regards Sebastian
> >>>>>>
> >>>>
> >>>> _______________________________________________
> >>>> Support-list mailing list
> >>>> Support-list at support.elphel.com
> >>>>
> >>>>
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >>>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Support-list mailing list
> >> Support-list at support.elphel.com
> >>
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >>
> >
> >
> > _______________________________________________
> > Support-list mailing list
> > Support-list at support.elphel.com
> >
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
> >
> >
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20111017/71f18a0b/attachment-0002.html>