[Elphel-support] Elphel 353 / 359 stereo camera problem
Simon Vogl
office at voxel.at
Tue Jun 23 04:44:43 PDT 2020
We're still trying without considerable progress; I switched on the
sdram sync debug bit in the /etc/init.d/fpga script, syslog reports a
number of errors, maybe they help to locate the issue:
Jan 1 00:00:40 Elphel353 kernel: Found MT9P001 2592x1944 sensor, chip ID=1801
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/framepars.c:420:initMultiPars GLOBALPARS(G_MULTI_NUM)=f
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:922:multisensor_pgm_detectsensor replaced: sensor->sensorDelay=0xed8
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1278:multisensor_pgm_sensorphase cableDelay1=0x708 FPGADelay1= 0xf3e sensorproc_phys->sensor.sensorDelay=0x99c, thisPhase1=0x9000f
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1283:multisensor_pgm_sensorphase cableDelay2=0x708 FPGADelay2= 0xefc sensorproc_phys->sensor.sensorDelay=0x99c, thisPhase3=0x9000c
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1288:multisensor_pgm_sensorphase cableDelay3=0x708 FPGADelay3= 0x101a sensorproc_phys->sensor.sensorDelay=0x99c, thisPhase3=0x90019
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x0 newPhase= 0x0 oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x0 newPhase= 0x0 oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0x93bc, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x1 newPhase= 0x0 oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0x3d13, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x2 newPhase= 0x0 oldPhase= 0x10000^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0xffff4835, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x3 newPhase= 0x0 oldPhase= 0x20000^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0xffffd37b, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0x0 oldPhase= 0x30000^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x3 newPhase= 0x7d oldPhase= 0x30000^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0xffffa2cb, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x3 newPhase= 0xbb oldPhase= 0x3007d^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xda oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xea oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xf2 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xf6 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xf8 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xf9 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1646:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0x0 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x3 newPhase= 0xff83 oldPhase= 0x30000^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1797:multisensor_memphase centroid=0x285a, OK=0
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=0 newPhase90=0x3 newPhase= 0xff45 oldPhase= 0x3ff83^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff26 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff16 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff0e oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff0a oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff08 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x2 resetDCM=1 newPhase90=0x3 newPhase= 0xff07 oldPhase= 0xfffffffc^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1673:multisensor_adjustSDRAM DCM error=4
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1685:multisensor_adjustSDRAM low90=3, low=250, high90=3, high=-250
Jan 1 00:00:40 Elphel353 kernel: **** ERROR adjusting SDRAM clock phase in arch/cris/arch-v32/drivers/elphel/multisensor.c:1697:multisensor_adjustSDRAM
Jan 1 00:00:40 Elphel353 kernel: low90=3, low_l=249, high90=3, high_l=-250
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1310:multisensor_pgm_sensorphase **** ERROR adjusting SDRAM clock phase in arch/cris/arch-v32/drivers/elphel/multisensor.c:1311:multisensor_pgm_sensorphase, result=0xffffffff
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x3 resetDCM=1 newPhase90=0x9 newPhase= 0xf oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x4 resetDCM=1 newPhase90=0x9 newPhase= 0xc oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x5 resetDCM=1 newPhase90=0x9 newPhase= 0x19 oldPhase= 0x0^M
Jan 1 00:00:40 Elphel353 kernel: Resetting MT9X001 sensor^M
Jan 1 00:00:40 Elphel353 kernel: Reading sensor registers to the shadows:^M
Jan 1 00:00:40 Elphel353 kernel: Initializing MT9X001 registers with default values:^M
Jan 1 00:00:40 Elphel353 kernel: Starting hardware sequencers
Jan 1 00:00:47 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x3 resetDCM=0 newPhase90=0x1 newPhase= 0xb0 oldPhase= 0x1000f^M
Jan 1 00:00:47 Elphel353 kernel: arch/cris/arch-v32/drivers/elphel/multisensor.c:1506:multisensor_set_phase reg_addr=0x4 resetDCM=0 newPhase90=0x1 newPhase= 0xb0 oldPhase= 0x1000c^M
HW needs replacement?
Simon
Am 22.06.20 um 20:35 schrieb Oleg:
> Yes, 8.2.15 should be fine too... as long as you find the MULTI_*
> parameters.
> Regards,
> Oleg
>
> On Mon, Jun 22, 2020 at 12:25 PM Simon Vogl <office at voxel.at
> <mailto:office at voxel.at>> wrote:
>
> Hi Oleg,
>
> thanks for the timely feedback, let me answer inline:
>
> Am 22.06.20 um 19:44 schrieb Oleg:
>> Hi,
>>
>> When stacking images vertically the mux board let's the 1st
>> sensor (top image) through w/o buffering in SDRAM. The second is
>> buffered. So, it could be SDRAM phase.
>>
> Well, this is definitely involved - when I set the right clock
> parameters, I could get reasonable picture content again (although
> only on one image).
>> 1. What firmware version are you using?
>> ~# cat /etc/issue
>> or
>> ~# cat /etc/release
>>
> [root at Elphel353 /root]817# cat /etc/release
> JFFSID="8.2.15"
>
> close to .16 but not exactly - does an upgrade in the field make
> sense?
>
>> 2. If the firmware is 8.2.16 (hopefully), then the phase settings
>> are available through the camera parameters:
>> http://192.168.0.9/autocampars.php -> check 'unsafe' - in the
>> table you will find:
>>
>> MULTI_PHASE_SDRAM
>> MULTI_PHASE1
>> MULTI_PHASE2
>> MULTI_PHASE3
>>
>> Check the values from a camera that works ok. And try to fix them
>> for a bad camera from this autocampars page (not
>> 10359_controls.html, which communicates directly with fpga -
>> cannot not store settings across reboots)
>> Once you set the values, go back to
>> http://192.168.0.9/autocampars.php and 'save' the config in the
>> second table. Then reboot.
>>
> ok great, will try it this way, guess that's what I was missing.
>> 3. Has anything changed recently? Like you stored a new
>> configuration? Reboot often? Applied any changes?
>> The camera stores and loads its firmware from flash -
>> sometimes the fs or some file there would get corrupted when
>> changes to the filesystem are applied but not properly synced.
>> With a single lens camera most of the trouble was usually
>> caused by /etc/autocampars.xml going corrupt. Then backing it up,
>> removing the file and then rebooting (it autoregenerated on
>> reboot if not there) might help. Note - if you manipulate files
>> manually from a command line - run 'sync' when done. Then repeat
>> step 2 if firmware is 8.2.16.
>>
> The maintenaince guys did a few reboots last week (suspected
> troubles with a poe switch) and I am hoping that this is the cause
> of the trouble.... I bit-compared the bit files, they're identical
> to a running camera. The xml files looked fine as well.
>
> Will try around more and come back to you - thanks a lot!
>
> Simon
>
>
>> Regards,
>> Oleg
>>
>> On Mon, Jun 22, 2020 at 10:29 AM Simon Vogl <office at voxel.at
>> <mailto:office at voxel.at>> wrote:
>>
>> Dear all,
>>
>> we have a bunch of 353-based stereo cameras that are in
>> operation for seven years meanwhile and we've been very happy
>> with them. Over the last weeks, some of them have image
>> errors, at first sight it looks like one of the sensor is not
>> configured correctly, a combined image looks like:
>>
>> I found that this seems to be a problem with the 10359
>> multiplexer board -- exchanging this resolves the issue for a
>> camera.
>>
>> When booting, the relevant portion of dmesg reports this
>> error on the affected cameras:
>>
>> arch/cris/arch-v32/drivers/elphel/framepars.c:390:initGlobalPars GLOBALPARS(G_DEBUG)=0
>> initSequencers:resetting both sequencers
>> arch/cris/arch-v32/drivers/elphel/framepars.c:420:initMultiPars GLOBALPARS(G_MULTI_NUM)=0
>> Initializing DMA registers for EXTDMA3 -
>> sensor clock set to 48000000
>> 10359 bitstream version =0x0359a054
>> 10353 sensor clock set to 96000000
>> 10359A sensor clock set to 96000000
>> removing MRST from the sensor
>> Found MT9P001 2592x1944 sensor on 10359A port 1, chip ID=1801
>> Found MT9P001 2592x1944 sensor on 10359A port 2, chip ID=1801
>> Setting internal HACT generation
>> Found MT9P001 2592x1944 sensor, chip ID=1801
>> arch/cris/arch-v32/drivers/elphel/framepars.c:420:initMultiPars GLOBALPARS(G_MULTI_NUM)=f
>> **** ERROR adjusting SDRAM clock phase in arch/cris/arch-v32/drivers/elphel/multisensor.c:1697:multisensor_adjustSDRAM
>> low90=3, low_l=249, high90=3, high_l=-250
>> arch/cris/arch-v32/drivers/elphel/multisensor.c:1310:multisensor_pgm_sensorphase **** ERROR adjusting SDRAM clock phase in arch/cris/arch-v32/drivers/elphel/multisensor.c:1311:multisensor_pgm_sensorphase, result=0xffffffff
>> Resetting MT9X001 sensor
>> Reading sensor registers to the shadows:
>> Initializing MT9X001 registers with default values:
>> Starting hardware sequencers
>> arch/cris/arch-v32/drivers/elphel/quantization_tables.c:211:init_qtable_head_cache
>>
>> I have tried to set the clock parameters from a functioning
>> specimen via the /359/10359_controls.html which temporarily
>> resolves the issue, but after a reboot the problems reappear.
>>
>> Any chance to resolve this in software or do we need to
>> exchange boards? If so - are they still available?
>>
>> All the best from Austria, stay healthy,
>>
>> Simon
>>
>>
>>
>>
>> --
>> VoXel Interaction Design |www.voxel.at <http://www.voxel.at>
>> DI Dr.techn. Simon Vogl |simon at voxel.at <mailto:simon at voxel.at>
>> Tomaschekweg 46 | +43 650 2323 555
>> A-4040 Linz - Austria |
>> Office address: Industriezeile 35, 4020 Linz (2nd floor)
>>
>> _______________________________________________
>> Support-list mailing list
>> Support-list at support.elphel.com
>> <mailto:Support-list at support.elphel.com>
>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>>
>>
>>
>> --
>> Best regards,
>> Oleg Dzhimiev
>> Electronics Engineer
>> phone: +1 801 783 5555 x124
>> Elphel, Inc.
>>
>> _______________________________________________
>> Support-list mailing list
>> Support-list at support.elphel.com <mailto:Support-list at support.elphel.com>
>> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
> --
> VoXel Interaction Design |www.voxel.at <http://www.voxel.at>
> DI Dr.techn. Simon Vogl |simon at voxel.at <mailto:simon at voxel.at>
> Tomaschekweg 46 | +43 650 2323 555
> A-4040 Linz - Austria |
> Office address: Industriezeile 35, 4020 Linz (2nd floor)
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> <mailto:Support-list at support.elphel.com>
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
>
>
>
> --
> Best regards,
> Oleg Dzhimiev
> Electronics Engineer
> phone: +1 801 783 5555 x124
> Elphel, Inc.
>
> _______________________________________________
> Support-list mailing list
> Support-list at support.elphel.com
> http://support.elphel.com/mailman/listinfo/support-list_support.elphel.com
--
VoXel Interaction Design | www.voxel.at
DI Dr.techn. Simon Vogl | simon at voxel.at
Tomaschekweg 46 | +43 650 2323 555
A-4040 Linz - Austria |
Office address: Industriezeile 35, 4020 Linz (2nd floor)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20200623/46f02ba6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obnfpilahfcneomg.png
Type: image/png
Size: 46589 bytes
Desc: not available
URL: <http://support.elphel.com/pipermail/support-list_support.elphel.com/attachments/20200623/46f02ba6/attachment-0001.png>
More information about the Support-list
mailing list